Development Guide

If JMdictDB was installed for production use — that is, there are users who expect the data in the default "jmdict" database to be consistent and preserved — but you want to do development work on the code using the same Postgresql server, you will want to create a dedicated development database to prevent changing the contents of the live, production "jmdict" database.

The development database should be treated as transient; delete and recreate as often as needed. It’s also sometimes useful to have multiple development databases, just remember their names have to be registered in the JMdictDB config-pvt.ini file in order to access them via the web UI. The command line tools all accepts a -d/--database or similar option to specify the database to access.

Below we will assume that the development database is named "jmdev"; you can name it whatever you wish as long as you:

If you are using the "jmdict" database for development, use that name below rather than "jmdev".

You can now create a development database by several means:

These are Apache directives that control the operation of the mod_wsgi component and will go in a location determined by the Apache web server. The "WSGIScriptAlias" directive tells Apache’s mod_wsgi component where to find the .wsgi file that is used to load the JMdictDB application code:

This file is only relevant when the JMdictDB app is being served via the Apache web server; it is not relevant when running the Flask server.

This is an importable Python file that is run by the Apache mod_wsgi component. It is commonly located (in a development environment) in the web/lib/ directory. It identifies the main configuration file that the JMdictDB Flask application will read when it starts, jmdictdb.ini below.

This file is only relevant when the JMdictDB app is being served via the Apache web server; it is not relevant when running the Flask server.

When the JMdictDB app is run via the Apache web server it reads the main configuration file given in the .wsgi file (jmdictdb.ini in example above). When run via the Flask development server, the config file to use is given on the command line used to run the server.

The main configuration file contains references to two other files:

Defines which databases are available through the web UI and provides access credentials for them.

Location and name of file that JMdictDB app logging messages will be written to.

This file contains credentials that the JMdictDB code will use to access the database. If there is an existing jmdictdb-pvt.ini file for access to the main "jmdict" database, you can give this file a different name (e.g., debug-pvt.ini) and set that name in the [config]/PRIVATE option in the main confuration file below.

Change to directory {{DEVDIR}}/web/lib/, copy jmdictdb-pvt.ini-sample to jmdictdb-pvt.ini and edit it:

If using a dedicated development database, uncomment the lines in the "[db_jmdev]" section. Change "jmdev" here and in the "dbname" name if the development database name if different.

Uncomment the [db_jmtest01] section and change the usernames and passwords as described above. Web access to the jmtest01 test database in needed by the web page tests in described in section 3.2, “Web page tests”.

If you want to prevent access by the web UI development code to the production database, remove the [db_jmdict] section entirely. Note though that this will not prevent access via the command line tools.

Make sure that jmdictdb-pvt.ini is not world-readable but is readable by the web server process, for example:

(You may need a user name other than "www-data" on some systems; often "httpd" on Redhat derived system for example.)

When running the Flask development server it is convenient to use a separate config file tailored to debugging with logging messages directed to the terminal rather than a log file and possibly different logging parameters. To avoid conflict with any existing jmdictdb.ini file, the config file for use with Flask can be named debug.ini.

Copy jmdictdb-sample.ini to debug.ini and edit it.

Change the [config]/PRIVATE option value if you are using a private config file named other than "jmdictdb-pvt.ini".

Change the default database service to that for the development database (if not using the jmdict database for development of a non-production machine.) E.g., change:

to

Comment out the "LOG_FILENAME = …​" line. When commented out, log output will go to the terminal window Flask is being run in.

You may also want to change the logging level or enable certain logging messages in the [logging]/LOG_FILTERS section.

You can serve the application code from a newly cloned JMdictDB directory or from an existing {{DEVDIR}} directory. We will refer to either as {{TESTDIR}} below.

Create a main configuration .ini file based on the sample file:

You can use some other name, it need not be jmdictdb.ini; its name will will be given in the .wsgi file ( 2.6.4, “Wsgi file”) or via an Apache environment variable.

Edit it and make similar changes as described above in 2.4.2, “Main configuation file (debug.ini)”

You can use the same jmdictdb-pvt.ini (or whatever name you used) file ( 2.4.1, “Private configuration file (jmdictdb-pvt.ini)”) to give the web server access to the same set of databases, or create an additional *-pvt.ini file with different databases and change the value of [config]/PRIVATE in the jmdictdb.ini file to point to it.

but point LOG_FILENAME to a real file (if left commented out messages will go to stderr which the web server may write to its own logs or discard.)

Be sure to create the log file as JMdictDB app will not create it automatically. It must be writable by the web server.

For more details on Apache configuration, see section 6.3. Configure the Apache webserver in the Installation Guide.

In this example the test/preview app will be made accessible at the URL https://jmdemo/. You can of course change "jmdemo" to anything you wish.

{{TESTDIR}} denotes the directory into which you cloned the JMdictDB code for the running the test/preview server.

Line 3 points to the .wsgi file which this example is placed in {{TESTDIR}}/lib/jmdemo.wsgi; you can use a different name or place it elsewhere and adjust that line to match if you wish.

The .wsgi file is very similar the one used for a production instance and described in section 6.2.4. Create a .wsgi file of the Installation Guide except:

The .wsgi file does not need to have execute permission but does need to be readable by the webserver process.

A test runner program, runtests.py, was written that provides more control of the tests run and the results format than the unittest module does (or did, when runtests.py was written.)

The command (run in python/tests/):

will run all library tests. To run a specific test or set of tests, specify them as 'module name', 'class' and 'test'. Examples:

By default tests are looked for it the tests/ subdirectory of the main tests/ directory but a different directory can be specified with runtests.py’s --dir option.

Tests can also be specified by filename path by including a "/" character in the test argument. When this is the case the --dir argument is ignored. To run the tests in the file "newtest.py" located in the main tests/ directory (not tests/tests/):

Leave off the ".py" when adding a test class or test method"

Multiple test arguments can be given. To debug test failures:

This will start the test under the Python debugger (type 'c' to stop running) and will stop at the first error or failure allowing the use of pdb to diagnose the problem. (Note that it will stop at any exception including intentional ones such as SkipTest so this is best used when running specific individual tests.)

For more details on argument syntax: ./runtest.py --help

To run all the tests:

or

The Selenium (https://selenium.dev) web driver will need to be installed on the development/test machine. The test code is currently hardwired to use Firefox as the target browser in the setUpModule() functions of the web test files.

The web tests need to access the jmtest01 database via the svc URL parameter; you will need to have a [db_jmtest01] section in the config-pvt.ini file used when running the Flask debug server. See sections 2.5, “The Flask development server” and 4, “The Flask development server” for details about setting up and running the Flask development server.

The web page tests assume the existence of a user JMdictDB editor account that the tests can log in as. A separate JDdictDB user is recommended for this purpose although an existing real user can be used if desired. If a dedicated test user account is used, it can be manually created one time with either the JMdictDB users.py web page or with the bin/users.py command line tool. (This assumes the typical case where both the production server’s and the Flask development server’s configuration files specify the same "jmsess" user database and thus share the same set of JMdictDB users.) The account userid, full name and password may be whatever you wish. They should be placed in a file named:

in the order given, one per line. Case must match. Unless all users on the machine are trusted, read access should be limited to the user who will be running the tests.

Start the Flask web server using the default port of 5000. (That port is currently hardwired into the test code.)

To run all the tests:

You can run individual tests the same way as for the library tests by specifying the test module / test class / test method names, "." separated, as arguments to runtests.py.

Test code should load a test database by importing DBmanager from module jmdb.py and calling DBmanager.use(DBNAME,DBFILE) where DBNAME is the name of a database to create in the Postgresql server and DBFILE is the name of a file containing a Postgresql database dump that are the contents of the data to load.

The first time the dump file is loaded by jmdb.DBmanager.use(), a hash of the dump file is calculated and saved in the database. When tests are run again later, the database hash is checked against hash of the request dump file and if the former does not match or does not exist, the database is reloaded from the dump file. If there is a match, the correct database is already loaded and can be used without a time consuming reload. This of course presumes that tests do not make any modifications to the database that would invalidate it for subsequent tests. If that is not the case the test should invalidate the database hash to force a reload the next times tests are run. This can be done by deleting the row(s) in testsrc or by updating testsrc.hash to an invalid value like ''.

The script ./load_testdb.sh can be used to unconditionally force a reload of the test database. This is useful, for example, before making and saving changes to the test database to assure the changes are made to a clean, unmodified version of the database.

load_testdb.sh is a simple script that effectively accomplishes the following:

but in addition saves a hash of the .sql file to a table in the database. This allows the tests to verify the correct version of jmtest02 is loaded and reload it if need be.

When new tests are added it is sometimes necessary to add new test entries to the test database(s). Additionally, if the JMdictDB database schema is updated to a new version, the test database must be updated to match.

Maintenance on the test database can be performed by loading into Postgresql and using the JMdictDB web or commandline tools or using the usual Postgresql tools (e.g. psql). If using the web pages to edit/add/delete entries you’ll need to add a "service" section to the active config.ini (or config_pvt.ini) file to make the test database accessible to the web UI via the svc=…​" url parameter.

To update the test database to the current JMdictDB database version:

Load a fresh copy of the test database:

Then apply the latest database updates to as you would to the main "jmdict" database:

And when all changes are complete, save it:

When the database schema or tables' contents is changed in a way that could result in pre-update JMdictDB code failing, a new update number is added and existing update number(s) deactivated. This effectively replaces the previous update number(s) with a new one and prevents older code that doesn’t understand the newer database changes from attempting to access it.

Some changes do not affect the ability of the pre-update version code to work with the post-update version of the database. Examples could be changes to tags that aren’t referenced in the code anywhere or the addition of database objects (tables, views, etc) that aren’t referenced in the code yet. In this case what we do is to add an additional update number. The earlier update number remains active and controls code access to the database but the additional number provides supplementary information that the tag update has been applied to the database.

This specifies the database update number that will go into the "db" table when a new JMdictDB database is created.

If this is a replacement update (see 5.5.1, “Replacement update” above), then find the line near the top of db/mktables.sql that looks like:

where "zzzzzz" is a 6-digit hexadecimal string, or possibly several such strings separated by commas, and replace everything inside the quotes with the new 'xxxxxx' update number.

If this is a supplementary update (see 5.5.2, “Suplementary update” above) and add "xxxxxx" to the end existing numbers:

If this is a replacement update (see 5.5.1, “Replacement update” above) then replace the the existing version number(s) with the new one(s). For example, rev 220826-7340b74 updated the database to version cda09c and replaced:

with:

If it is a supplementary update (see 5.5.2, “Suplementary update” above) then no change to dbvers.py is needed; by definition a supplementary database update adds features to the database that will not prevent the existing version of the code from working correctly thus the code need only check for the non-update version of the database.

First, load a fresh, clean copy of the test database:

Apply the database update to it:

If there were errors applying the update, fix them and repeat the above steps including reloading the test database from the jmtest01.sql file.

After the update applies cleanly, save a temporary copy of the updated jmtest01 database. It is important to do this before running any tests as some tests may modify the database.

Run the tests again:

If any errors occur in the tests the problem may be due to a problem in the database update script (in which case correct it and continue from the start of this section), or some tests themselves may need updating to work with the updated database. In the latter case, fix the tests until all pass.