Some plugins aren't being mirrored

Take this plugin for example: http://www.pluginmirror.com/plugins/wp-post-date-remover

The plugin is still hosted and functional. A github link is even created for it, but it hasn't been cloned.

We just re-added this to the queue and it is currently about 10th in line. Please let me know if it is not updated within the next hour or so.

Just a quick update. It is still in the queue but it is working its way to the top.

Looks like the repo was created, but nothing got put in it.

Think it could be because there's nothing in trunk because there's a
release in the tag tree?

There's a reason I prefer git when it comes to pulling plugins. This
service is a godsend for keeping wordpress plugins updated. Thanks for the
help thus far.

On Fri, Jan 30, 2015 at 2:17 PM, Mike Hansen notifications@github.com
wrote:

Just a quick update. It is still in the queue but it is working its way to
the top.

—
Reply to this email directly or view it on GitHub
#4 (comment).

http://www.pluginmirror.com/plugins/wordpress-importer also empty. We were just looking at this at The Loop stackexchange WP chatroom today.

Exactly, when this reached the top of the queue and was mirrored there was no content in the trunk directory so nothing was copied over.

See I came into this ass-backwards.

My use case is basically an infrastructure that maintains a large amount of
Wordpress blogs. Current count is about 175.

The only way to handle this in a sane fashion is some value of
configuration management. I use puppet (with some serious magic) to deploy
Wordpress sites, and to keep themes and plugins updated.

I came to discover, without knowing about the pluginmirror project, that
all the wordpress plugins were being mirrored on github. Which rules,
because subversion really sucks to clone against with puppet.

This is the only real outlier. I might just have to shove it into one of my
static content repos because it hasn't changed for ages (though it works
fine).

On Sat, Jan 31, 2015 at 12:15 PM, Mike Hansen notifications@github.com
wrote:

Exactly, when this reached the top of the queue and was mirrored there was
no content in the trunk directory so nothing was copied over.

—
Reply to this email directly or view it on GitHub
#4 (comment).

@MikeHansenMe

Hello @MikeHansenMe ! Same issue here. It seems repositories are, at some points, automatically emptied. It happened to me lastly with https://github.com/wp-plugins/google-sitemap-generator Though, it wasn't when I added it as a submodule couple days ago.
Is there a way to avoid this ? Thx

@xavismeh

Hi @xavismeh thanks for letting us know. We just cloned this. It should all be available now.

For future comments, before mentioning your plugin here. Let's check over at pluginmirror.com that it has been cloned. If not click clone, then let us know if it is not cloned after a few hours.

This one is also empty: http://pluginmirror.com/plugins/events-manager https://github.com/wp-plugins/events-manager/

My confusion is mostly that this was working in the past, at least with the
plugin I complained about.

Then it looks like the entire github repo got wiped, and things started
getting put back in.

But subversion repos with a tagged release but not a trunk release are no
longer making the cut.

On Mon, Feb 9, 2015 at 12:17 AM, Mitar notifications@github.com wrote:

This one is also empty: http://pluginmirror.com/plugins/events-manager
https://github.com/wp-plugins/events-manager/

—
Reply to this email directly or view it on GitHub
#4 (comment).

@mitar

@mitar thanks for letting us know. We will get it in the queue today.

@jowrjowr You are correct, the whole repo was deleted and we are still working to get everything restored. Currently there are around 500 plugins that the mirror thinks are already mirrored however they are not. The only way for us to get these ones pushed is manually. If you have another plugin that is not showing please leave a link here and we will get it added.

@mitar

That's the only one that I've personally found to be an issue, however
would it be possible to treat the internationalization repo as a wordpress
plugin as well?

http://i18n.svn.wordpress.org/

Keeping this updated on the client side is a massive nuisance because of
its' size, and it is really important for non-english blogs.

On Mon, Feb 9, 2015 at 8:32 AM, Mike Hansen notifications@github.com
wrote:

@mitar https://github.com/mitar thanks for letting us know. I will get
it in the queue today.

@jowrjowr https://github.com/jowrjowr You are correct, the whole repo
was deleted and we are still working to get everything restored. Currently
there are around 500 plugins that the mirror thinks are already mirrored
however they are not. The only way for us to get these ones pushed is
manually. If you have another plugin that is not showing please leave a
link here and we will get it added.

—
Reply to this email directly or view it on GitHub
#4 (comment).

@MikeHansenMe

Hello @MikeHansenMe
I'm here to bring the bad news ! Here are some more plugins that have empty repository on Github :
intensedebate
multi-varnish-http-purge
permalink-finder
stats
wp-db-driver
wpmandrill
wp-polls
yet-another-related-posts-plugin

Thanks

@xavismeh

Thanks @xavismeh, I appreciate your effort here. We wil get them queued.

@jowrjowr Currently the Plugin Mirror cannot do other repos but it is something we could look into.

@xavismeh

It looks like I was mistaken when I said there weren't any others.

cachify
simple-full-screen-background-image
simple-google-news
bing-translator
efficient-related-posts
wordpress-23-related-posts-plugin
yet-another-related-posts-plugin
category-posts-in-custom-menu
remove-open-sans-font-from-wp-core

I am an edge case magnet, I swear go god!

These appear to all have non-empty subversion trunks, too.

These plugins were all properly cloning from the pluginmirror github in the
past, unlike my other mentions. I'm not precisely sure when they stopped
working - my logs on this server don't go back beyond the 29th, and I only
noticed today because the server that this stuff is getting is now coming
online and I'm looking more closely.

It is, by the way, much appreciated that you are keeping these mirrored
into github. Subversion is just not nice to work with, at least with puppet
and repo cloning/updating.

Might I suggest a "This is broken, rekick" button on a plugin's page? That
way folks don't have to manually report isues like this.

On Tue, Feb 10, 2015 at 8:46 AM, Mike Hansen notifications@github.com
wrote:

Thanks @xavismeh https://github.com/xavismeh, I appreciate your effort
here. We wil get them queued.

@jowrjowr https://github.com/jowrjowr Currently the Plugin Mirror
cannot do other repos but it is something we could look into.

—
Reply to this email directly or view it on GitHub
#4 (comment).

We added the plugins to the queue to be updated and also added a new button "Update" to force an update. You can now find plugins on plugin mirror and click update. It will still have to go throught the queue but at least users can trigger it. Thanks again for your effort and feedback.

@MikeHansenMe

Where is the button? @MikeHansenMe 😃
http://www.pluginmirror.com/plugins/soundcloud-is-gold
Thank you!

Not sure why the button was not showing on that one but it has been pushed out now.

@MikeHansenMe

@MikeHansenMe I am seeing a lot of the plugins are empty lately.

In order to understand what causing the issue I tried to setup local copy of the project. I have managed to run the app, however I am not sure how the plugins actually get added to the pluginmirror app. So far looking at the code I am did not find much.

Am I missing something obvious here?

Really appreciate your help.

@azizur

Hey @azizur the empty plugins is not a result of the mirror. What happened was during the restore process all of the repos were created in a manual way and some of the plugins (~700) think they have been cloned on the mirror and link to github to empty directories. If the link exists on the site but the repo is not on github it is one of the 700 plugins that are in a state of limbo. For each of those we have to push them manually. If you have some specific plugins, please let us know and we will get them updated.

media-category-library is another one:
http://www.pluginmirror.com/plugins/media-category-library
with an empty github repo
https://github.com/wp-plugins/media-category-library/

Thanks for all the effort!

I wrote a small python script which automatically checks if a plugin is empty (on the wordpress site), has a github clone that is empty or not

import requests
import json
import sys
import time
import calendar

#https://api.github.com/rate_limit
def check_rate( requests, s):
        rate_limit_url = "https://api.github.com/rate_limit"
        rate_limit = s.get(rate_limit_url).json()
        # the "rate" rate limit is the smaller one
        if rate_limit["rate"]["limit"] < requests:
                print "number requests exceeds limits: " + str(requests) + ">" + str(rate_limit["rate"]["limit"])
                exit(1)
        # check if the remaining rate limit is sufficient
        if rate_limit["rate"]["remaining"] < requests:
                wait = rate_limit["rate"]["reset"] - int(calendar.timegm(time.gmtime()))
                if wait > 0:
                        print "wait " + str(wait) + " secs for the rate limit to reset"
                        time.sleep(wait)
        # now check if the resource limit is sufficient
        rate_limit = s.get(rate_limit_url).json()
        if rate_limit["resources"]["core"]["limit"] < requests:
                print "number requests exceeds limits: " + str(requests) + ">" + str(rate_limit["resources"]["core"]["limit"])
                exit(1)
        # check if the remaining limit is sufficient
        if rate_limit["resources"]["core"]["remaining"] < requests:
                wait = rate_limit["resources"]["core"]["reset"] - int(calendar.timegm(time.gmtime()))
                if wait > 0:
                        print "wait " + str(wait) + " secs for the resource limit to reset"
                        time.sleep(wait)
        print "rate remaining: " + str(rate_limit["rate"]["remaining"])
        print "resource core remaining: " + str(rate_limit["resources"]["core"]["remaining"])

def does_wordpress_plugin_exist(name):
        base_url = "https://wordpress.org/plugins/"
        plugin_head = requests.head(base_url+name)
        return plugin_head.status_code == 302
per_page = 50
all_repos_first_url = "https://api.github.com/users/wp-plugins/repos?per_page=" + str(per_page)
all_repos_url = all_repos_first_url + "&page=1"

# start a session
s = requests.Session()
# authenticate, which increases the rate limit
# s.auth = ('username','password')

while True:
        # check if rate allows the number of request (plus a safety margin)
        check_rate( per_page + 5, s)
        # get the list of repos for the current page
        repos = s.get(all_repos_url)
        content = repos.json()
        # iterate through all current repose
        for repo in content:
                if not does_wordpress_plugin_exist(repo['name']):
                        print "empty plugin: " + repo['name']
                        continue
                # get the commits
                commits_url = "https://api.github.com/repos/wp-plugins/" + repo['name'] + "/commits"
                commit = s.get(commits_url)
                commit_content = commit.json()
                # if there are no commits
                if "message" in commit_content and commit_content["message"] == "Git Repository is empty.":
                        print "empty repo: " + repo['name']
                else:
                        print "ok    repo: " + repo['name']
        # get the next page
        if "next" in repos.links:
                all_repos_url = repos.links["next"]["url"]
        else:
                break
        sys.stdout.flush()

there is a rate limit on the github api, so the 's.auth' should be used to authenticate. There is still a lmit of 5000 request (per hour I think), but the script will wait....

The output can be grepped for

"empty repo", meaning there is no github clone available - so they should probably be put into the cloning queue
"ok repo" which means, there is a clone available - so no need to do anything
"empty plugin" which means, there is a svn repo folder at wordpress, which is empty - probably because it was deleted (this is a completely different discussion as I read in issues 2 and 3) - I am not sure how git-svn handles those, since everything in the repo is gone - for git this also means the branches and tags would be deleted. So I guess there is no point in mirroring?! Or at least they do not need to be in any cloning queue...

The script finally ran through and unfortunately, it identified 22729 repos that are empty on github...
I think there are many plugins, that are not maintained anymore and might not even work - but I guess that was not the intention of the mirror to exclude those.

I'd love to see the public plugins all listed with non-empty repos. In my case, I don't know in advance which plugin I'll have to clone and process, and I want to minimize manual interaction to the bare minimum.
What can we do to help, would a list of all empty github repos help?

FYI: I got 33162 of 52712 repos reported as empty. Some come with a warning about being unlisted, others not.

Cannot attach the list, so here it is (too big for pastebin): http://pasted.co/97d7173d

We're working on the issues currently. There is no need for more examples at this time. We'll have a fix out as soon as possible.

Old version available: https://github.com/wp-plugins/wp-fullcalendar There is already 1.0 out.

Navigation Menu

Repository navigation

Some plugins aren't being mirrored #4

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Some plugins aren't being mirrored #4

Description

Activity

MikeHansenMe commented on Jan 30, 2015

MikeHansenMe commented on Jan 30, 2015

jowrjowr commented on Jan 31, 2015

lkraav commented on Jan 31, 2015

MikeHansenMe commented on Jan 31, 2015

jowrjowr commented on Jan 31, 2015

xavismeh commented on Feb 6, 2015

MikeHansenMe commented on Feb 6, 2015

MikeHansenMe commented on Feb 6, 2015

mitar commented on Feb 9, 2015

jowrjowr commented on Feb 9, 2015

MikeHansenMe commented on Feb 9, 2015

jowrjowr commented on Feb 9, 2015

xavismeh commented on Feb 10, 2015

MikeHansenMe commented on Feb 10, 2015

jowrjowr commented on Feb 11, 2015

MikeHansenMe commented on Feb 11, 2015

ajsb85 commented on Feb 14, 2015

MikeHansenMe commented on Feb 17, 2015

azizur commented on Mar 23, 2015

MikeHansenMe commented on Mar 23, 2015

nwoetzel commented on Jun 9, 2015

nwoetzel commented on Jun 9, 2015

nwoetzel commented on Jun 11, 2015

jhard commented on Oct 25, 2015

jhard commented on Oct 26, 2015

voldemortensen commented on Oct 26, 2015

mitar commented on Nov 18, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions