MyWiki:ReFill/technical

This page documents the architecture of reFill.

Components

Toolbar gadget

ReFillLink.js is a gadget that provides a way to submit a page to reFill in a single click.

Front end

This is a Vue.js single-page application where the user can enter the wiki page to process, select options, submit the page to the reFill API and check the results.

The front end code is bundled using webpack.

Middle layer

Celery is use to queue requests. Key-value data is held in memory using the Redis data store.

Back end

The back end code is written in Python and uses the Flask web framework. Docker provides containerization. Citoid is the main parser, called from citoid.py. Citoid is maintained by the Wikimedia Foundation and uses Zotero. The relationship between Citoid and Zotero is described in a tech talk.

Rather than reFill containing code to cope with the vagaries of thousands of different websites, Zotero translators can be created, benefiting other referencing tools used on Wikipedia.

Hosting

The production Docker container is hosted on the Toolforge PaaS environment, which uses Kubernetes.

• https://refill.toolforge.org/ng production front end

contents

/mnt/nfs/labstore-tools-project/refill

config
  /banners.php
  /citoid.php
  /common.php
  /README.md
  /stable.php
  /test.php
Git
  /date
    /src
      /Lang
      /Date.php
      /DateServiceProvider.php
    /tests
      /AutomaticTest.php
      /DateTest.php
      /TranslationKaTest.php
      /TranslationTest.php
    /composer.json
    /generator.php
    /LICENSE
    /phpunit.xml
    /README.md
logs [EMPTY]
old-ng [EMPTY]
refill-ng-dist [EMPTY]
versions
    /citoid
      /backend
      /bower_components
        /bootstrap
        /chosen
        /jquery
      /config
        /config.php
      /docker
        /refill-api
          /Dockerfile
        /refill-worker
          /Dockerfile
      /messages
      /src
        /Intuition
      /vendor
      /web
      /build-images.sh
      /LICENSE.md
      /Makefile
      /README.md
      /start.sh
    /stable
      /config
      /messages
      /src
      /templates
      /tests
        /Reflinks
      /vendor
      /web
        /components
        /hooks
        /ng
        /ng.old
        /scripts
        /style
        /acknowledgments.php
        /api.php
        /index.php
        /index_old.html
        /index_old.php
        /result.php
      /bower.json
      /composer.json
      /LICENSE
      /phpunit.xml.dist
      /Procfile
      /README.md
      /toolinfo.json
    /test
dist.tar.gz
index.html
README
refill2.tgz
replica.my.cnf
service.log
service.manifest
service.template
stats.php
test.php
uniqua.json

• https://refill-api.toolforge.org/ production api

contents

/mnt/nfs/labstore-tools-project/refill-api

 /archived
    /refill.yml
    /probe.sh
    /run-probe.sh
    /refill.yaml.old
    /refill-new.yaml
    /deployment.yml
 /refill
    /backend
       /apicache-py3
       /__pycache__
          /celeryconfig.cpython-37.pyc
          /celeryconfig.cpython-34.pyc
       /refill
          /utils
          /transforms
          /dataparsers
          /__pycache__
          /models
          /formatters
          /tasks.py
          /throttle.ctrl
          /__init__.py
       /throttle.ctrl
       /celerybeat-schedule.db
       /celeryconfig.py
       /requirements.txt
       /Pipfile.lock
       /Pipfile
       /app.py
       /pywikibot.lwp
       /celeryconfig.example.py
       /celeryconfig.docker.py
    /web
       /src
       /libs
       /webpack.config.js
       /package-lock.json
       /package.json
       /config.development.js
       /yarn.lock
       /template.html
       /README.md [contains build instructions for prod]
    /messages
    /docker
       /refill-worker
          /Dockerfile
       /refill-api
          /Dockerfile
    /refill
    /apicache
    /__pycache__
       /celeryconfig.cpython-34.pyc
    /worker-logs.txt
    /README.md
    /Makefile
    /start.sh
    /build-images.sh
    /nohup.out
    /throttle.ctrl
    /LICENSE.md
 /www
    /python
       /venv
          /bin
          /share
          /lib
             /python3.7
                /site-packages
          /include
          /pyvenv.cfg
 /logs [empty]
 /uwsgi.log
 /worker-deployment.yml
 /scheduler.sh
 /service.manifest
 /worker.sh
 /service.template
 /requirements.txt (10/1/2019)
 /replica.my.cnf

• https://refill-api-test.toolforge.org/ for test use. Probably inactive. Only Zhaofeng Li has access.

contents

/mnt/nfs/labstore-tools-project/refill-api-test

 /logs [EMPTY]
 /refill
   /backend
     /__pycache__
     /apicache-py3
     /refill
       /__pycache__
       /dataparsers
          /__pycache__
          /__init__.py
          /archiveis.py
          /arxiv.py
          /badauthors.py
          /baddates.py
          /citoid.py
          /detectdeadlink.py
          /doi.py
          /download.py
          /jstor.py
          /ncbi.py
          /parsespecial.py
          /parseurl.py
          /README.md
          /ris.py
          /title.py
       /formatters
          /__pycache__
          /__init__.py
          /citetemplate.py
          /formatter.py
          /plainlink.py
       /models
          /__pycache__
          /__init__.py
          /citation.py
          /context.py
       /transforms
          /__pycache__
          /__init__.py
          /fillexternal.py
          /fillref.py
          /mergeref.py
          /transform.py
       /utils
          /__pycache__
          /__init__.py
          /errors.py
          /parser.py
          /test_parser.py
          /utils.py
       /__init__.py
       /tasks.py
       /throttle.ctrl
     /app.py
     /celeryconfig.docker.py
     /celeryconfig.example.py
     /celeryconfig.py
     /Pipfile
     /Pipfile.lock
     /pywikibot.lwp
     /requirements.txt
     /throttle.ctl
   /docker
       /refill-worker
          /Dockerfile
       /refill-api
          /Dockerfile
   /messages
   /web
     /libs
        /wdiff.js
     /src
        /components
          /Change.vue
          /ChangeDetails.vue
          /ErrorDetails.vue
          /PreferencesEditor.vue
        /pages
          /Index.vue
          /LegacyBridge.vue
          /PageNotFound.vue
          /Result.vue
        /App.vue
        /main.js
        /Store.js
        /Utils.js
     /config.development.js
     /package.json
     /package-lock.json
     /README.md
     /template.html
     /webpack.config.js
     /yarn.lock
   /build-images.sh
   /LICENSE.md
   /Makefile
   /README.md
   /start.sh
 /www
   /python
      /venv
 /replica.my.cnf
 /service.manifest
 /uwsgi.log
 /worker.sh
 /worker-deployment.xml

Access to the hosting environment is via PuTTY and WinSCP - PuTTY for SSH command line access, and WinSCP for file transfer.

How to's

• How to set up a local instance of reFill on Windows
• How to set up an instance of reFill using the Windows Subsystem for Linux, or a virtual machine on Azure
• How to restart reFill when it is stuck

Discussions and links

• Wikipedia:Village_pump_(proposals)#RfC:_Block_reFill_tool_until_fixed (opened 2022 January 21)
• https://github.com/CurbSafeCharmer/refill, GitHub repo for ReFill
• User:Zhaofeng Li, original creator of tool
• User:Curb Safe Charmer, one of the current maintainers
• Search for "ReFill" in all village pumps
• User talk:JPxG/Archive16#reFill
• User talk:TheresNoTime/Archive 11#ReFill
• meta:Community Wishlist Survey 2022/Citations/New reference-filling tool
• Phabricator discussion re Citoid support by WMF
• Citoid workboard
• MfD re shutting down ReferenceExpander
• List of refs expanded by ReferenceExpander that need checking