Grants:IEG/Batch uploader for small GLAM projects

This project is funded by an Individual Engagement Grant

statusselected

Individual Engagement Grants

Batch uploader for small GLAM projects

summaryWe want to create a simple-to-use batch uploader for small-to-medium entry level partner institutions.

_targetWikimedia Commons

strategic priorityimproving quality

themetools

amount2175 USD

grantee• Yarl

advisor• Polimerek

contact• pawel.marynowski@wikimedia.pl

volunteer• Marta Malina Moraczewska

this project needs...

volunteer

give feedback

join

endorse

created on02:15, 29 September 2015 (UTC)

round 2 2015

Friendly space expectations

Project idea

What is the problem you're trying to solve?

It is frequently the case that small-to-medium GLAM institutions have digitized a part of their collections, but making the content available online is problematic for them: many museums haven't got the infrastructure enabling them to create an online catalogue from which the public domain images would be downloadable. As a result the websites of many institutions (museums of archaeology, natural history, smaller local museums and so on) feature only a handful of images, while the collections are not available. Frequently these small GLAMS are interested in collaborating with Wikimedia - this way the institution gains a place to make their collection available online.

However, the initial options these institutions are offered is either to use the Commons' Upload Wizard, or to set up the somewhat challenging GLAMWiki Toolset. Neither of these tools is perfect for the scale of planned uploads and the GLAM capabilities. The Upload Wizard does not offer an easy way to fill in the fields required by most GLAMs at the stage of upload (i.e. fields from the Artwork template). Also, Commonist and VicuniaUploader are not well suited for this task, as it requires manual rewriting the information from GLAM databases to the program interfaces which requires a lot of redundant work.

Summary of current alternatives
Name	Comment
Upload Wizard	Not designed for GLAM images. Really hard to use `{{Artwork}}` or `{{Photograph}}` template
Commonist / VicunaUploader	Designed mainly for self-made photographs. Technically it's possible to upload non-photographic information, but it's rather hack.
GLAMWiki Toolset	Not suitable for small GLAMs, preparation of upload is time consuming and requires prepared metadata

Worth mentioning is ComeOn!, French alternative. ~~Unfortunately due to software bug it's not possible to run it on Polish language OS.~~ Fixed in the meantime.

What is your solution?

We would like to create a simple tool that will enable the volunteer or employee working in a GLAM institution to simplify choosing and uploading selected files to Wikimedia Commons. The tool will generate a spreadsheet in a folder containing selected files to be uploaded. The spreadsheet will contain a list of filenames in that folder, as well as a list of Artwork field values as column headers. The users - GLAM volunteers or employees - can then manually (or using semi-automated database converters available in many office software packages) add metadata/information about these files (eg. author, date of creation, accession number, etc.) in each column of the spreadsheet. Each piece of information will be stored in a separate column, and before upload will be converted into wikicode by the upload tool. The advantage of such a tool will be let GLAM volunteers or employees use the office software packages of their own choice (Calc, Excel etc..). The only requirement will be to be able to export final spreadsheet in Excel or OpenOffice format.

Project goals

The goal of this project is to offer many small institutions collaborating with Wikimedia chapters simple software which will be usable regardless of the institution's collection infrastructure:

easy to understand and use;
an upload of a batch of files can be made faster than using the Commons' Upload Wizard;
allowing the GLAM (or other institution) more instant control over file descriptions than the Upload Wizard
focusing on file description - an advantage for the institution and ensuring quality of description
flexibility in choosing needed fields from the artwork template.

Technical details: program will be written using Java language. Code will be released on GitHub platform under GNU GPL license.

Project plan

Activities

Stage one

The project will start in January and conclude at the end of April 2016.

In the first month we will:

recruit a team of volunteer small GLAM organisations (as diverse as possible) to help assess and test the tool
conduct a survey among the GLAM volunteers, describing our application and featuring questions about any initial requests or needs of the GLAM volunteers and staff related to the application, in the context of the GLAM submitting images to Wikimedia Commons (for example, the need to incorporate templates other than Artwork into the application).

Stage two

The second month will be devoted to:

programming and developing the batch uploader,
preparing a user manual in English
preparing a community site for the application and the manual.

In the third month the test version will be sent to the 20 volunteers to perform a controlled upload and report back on usability, any technical problems or errors or needed improvements.

Stage three

In the fourth month:

the reported problems or requests will be addressed and added or corrected in the application.
the working version of the application will be announced on relevant Wikimedia community channels.

Budget

Funding is only needed for the work of the Project Leader (Yarl). Other work (recruiting GLAM volunteers, dissemination) will be done by means of Wikimedia volunteers and GLAM coordination. [Optionally, additional limited funds may be needed for project dissemination].

Programming work:

For month 2 (see Activities) - full time programming (1 month): 1450 USD
For month 4 - part time programming (1 month): 725 USD

Total Budget: 2175 USD

Community engagement

We will keep in touch and engage relevant communities (see below) via an on-Wiki projectpage, and the software's page on GitHub. GLAM institutions will be notified via mailing lists and community sites such as openGLAM.org. The communities which may be interested in commenting and collaborating include:

the Wikimedia Commons community,
Wikipedians;
GLAM-related communities;
GLAM volunteers and coordination;
Open GLAM organizations and activists.

We will provide opportunities to participate/comment on the community pages, the project page, social media; the project leader will also be available for a monthly hangout.

Sustainability

The batch uploader is meant as a tool for entry-level GLAM collaborations and as such it can be used by the increasing number of institutions interested in submitting quality content to Wikimedia Commons.
The software's open license allows users to build upon the basic version.

Measures of success

The batch uploader tool will be available in a stable version and functional by the end of the project;
At least 5 new GLAM institutions will contribute content to Wikimedia Commons;
Qualitative data will be gathered from participating GLAM institutions on the usability of the tool versus pre-existing tools:
1. Survey sent to participating GLAM volunteers,
2. Online survey open to all interested parties;
At least 10 GLAM institutions using the application a year after the tool release;
At least 300 image files uploaded by GLAM volunteers by end of project & 50 articles illustrated;
The number of files contributed to Wikimedia Commons will be counted via a Commons category related to the tool.

Get involved

Participants

Yarl – I'm a long-term Wikimedian. I have been working as webdeveloper since 2014, but besides that I've done some coding using Java language. I'm the author of VicuñaUploader, a popular Wikimedia Commons upload tool. I also help Polish chapter in GLAM cooperation by writing small upload scripts based on the needs of a specific GLAM project [1].
Marta Malina Moraczewska – I intend to volunteer on this project, as it in my opinion would considerably widen the group of small GLAM institutions collaborating with Wikimedia and eliminate some problems at entry level. I'm also the GLAM-Wiki coordinator for Wikimedia Poland, which would allow me to more easily enlist GLAM institutions (from at least 3 different countries) to help test and implement the tool.

Community Notification

Wikimedia Commons:

Village Pump

GLAM mailing lists:

glam@lists.wikimedia.org
cultural-partners@wikimedia.ch

Other:

wikipl-l@lists.wikimedia.org

Endorsements

Do you think this project should be selected for an Individual Engagement Grant? Please add your name and rationale for endorsing this project in the list below. (Other constructive feedback is welcome on the talk page of this proposal).

I have worked with Yarl on Vicuña Uploader and he is very good at getting the code up and running. He has a good feeling of the community needs and is capable of translating them into a working code fast and without too much fuss. I would be also happy to assist this project as well. « Saper // talk » 10:18, 30 September 2015 (UTC)
I think is an useful idea. I have no experience with GLAM project but I understand the need for specific tool.--Alexmar983 (talk) 22:53, 1 October 2015 (UTC)
I think having more competition in this area would be a good thing. Bawolff (talk) 16:33, 3 October 2015 (UTC)
Yarl's VicuñaUploader is the tool, and his experience with tools such as BursztaBot to upload Burszta Archives material to Commons are definitely strong points for endorsing the idea already. Yarl's involvement and skills are an added bonus to my enthusiasm as far as this project is concerned. Wojciech Pędzich Talk 18:14, 5 October 2015 (UTC)
Quite often our community forgets how difficult our magic is to the outside world. No, wikicode is not simple. No, uploading files to Commons is not easy. No, you can't learn it overnight. Now seriously, any tool helping to bridge a technological gap between us and our partners would be highly appreciated. Great idea, two thumbs up. Halibutt (talk) 20:32, 6 October 2015 (UTC)
I've used VicuñaUploader and find it a really practicable and user-friendly tool. In addition, Yarl's coding skills, commitment and enthusiasm for the project are a valuable asset not to be overlooked. Maire ♣ 20:38, 6 October 2015 (UTC)
From perspective of Wikimedia Polska GLAM cooperation, such a tool is a big need, as we have 10 open GLAM projects, and in 4 of them there is real need for such tool as the currently available have some unsolvable problems. Of course once created, such a tool can be useful for any mid-size GLAM coooperations. Polimerek (talk) 13:05, 8 October 2015 (UTC)
I and my colleague Spasimir who are working on the GLAM collaboration with Bulgarian Archives will definitely be interested to be testers and users of that software. →Spiritia 16:00, 8 October 2015 (UTC)
Support Anything that might tempt our GLAM partners to share content on Commons is a great idea. Theoretically everything they have in their collections is notable enough for its own article, so it would be nice to enable them to upload their own content without getting them file by file over the years from various Commons uploaders with varying degrees of accuracy in file names/titles/metadata. --Jane023 (talk) 15:23, 25 October 2015 (UTC)
Support I have been waiting for this tool since I started working with Wikimedia, none of the existing tools fulfill this role. I trust Yarl to make this happen, I'm a keen user of VicuniaUploader. I would be very happy to help any way I can. John Cummings (talk) 09:31, 2 December 2015 (UTC)
My personal experience in small grants shows that such a tool is very much in need! Alleycat80 (talk) 09:46, 2 December 2015 (UTC)
Even in a larger GLAM i am finding it difficult to access resources for large Toolset uploads, this tool would be a great benefit to the Wikipedian in residence here and at GLAM's around the world Jason.nlw (talk) 09:56, 2 December 2015 (UTC)
Yes! Please! This would be amazing! It would transform work with not just small GLAMs but larger organisations that are keen on making small batches of releases that synchronise with their internal photography/exhibition or events schedules. GLAMwiki needs this tool. PatHadley (talk) 09:56, 2 December 2015 (UTC)

It seems a very nice idea, especially after the "failure" (?) of the GLAMWIKIToolset Aubrey (talk) 10:24, 2 December 2015 (UTC)
The use case for this tool is very real and it would be so great to have this! I have worked on several such smaller uploads of sets of a few hundreds of images from GLAMs - too many to do by hand, many different creators for the images so not suitable for Commonist/Vicuna, and the institution can't provide an xml file for the GLAMwiki Toolset. The proposed spreadsheet-driven solution sounds perfect to me. Ah! I wish this existed already! Spinster (talk) 10:26, 2 December 2015 (UTC)
- And all this for only 2175 USD? That's a steal. Let's do this. Spinster (talk) 10:28, 2 December 2015 (UTC)
Support - Wonderful. I very much support this proposal. Husky (talk) 10:29, 2 December 2015 (UTC)
Support Such a tool would be very useful for our work with GLAM institutions. --Nicolas Rück (WMDE) (talk) 11:11, 2 December 2015 (UTC)
Support Very useful and necessary for GLAM institutions. Raymond (talk) 13:03, 2 December 2015 (UTC)
Support i have been praising vicunauploader at wikiconfusa, this fills big gap. Slowking4 (talk) 14:35, 2 December 2015 (UTC)
Terrific idea, very low price. PKM (talk) 18:16, 2 December 2015 (UTC)
Support I have used both GWtoolset and Vicuna uploader, and often thought that a more polished version of Vicuna could really help with smaller uploads, even at larger institutions. 85jesse (talk) 10:08, 3 December 2015 (UTC)
Support sounds ok. --Steinsplitter (talk) 15:07, 5 December 2015 (UTC)
It's a basic practical need for medium-scale GLAM projects. Drianmcdonald (talk) 22:30, 31 January 2016 (UTC)
Very useful indeed for small GLAMs Hansmuller (talk) 09:54, 28 April 2016 (UTC)
Support Vera (talk) 09:08, 30 June 2016 (UTC)
Pattypan has very quickly become my favorite tool for uploading files. It bridges a bridge between the various way data can be structured elsewhere and how we like meta data to be structured on Wikimedia Commons Vera (talk) 08:56, 26 January 2017 (UTC)
Have used Pattypan a number of times for batch-uploading audio recordings for Wikitionary. It works absolutely perfectly! (The Yes/Const/No bit might be confusing at first though) Prateek Pattanaik (talk) 16:48, 7 March 2017 (UTC)