Sandpoints supports a triadic hierarchy. This follows that content is organized, or parented according to a tree structure with a depth of three (entities). This project follows the scheme journal>issue>article (other examples to visualize this content are book>chapter>section, and ship>deck>room).
Note that the names of the basic triad elements are arbitrary. However:
The concept of backlinks was described in early speculations that informed hypertext (i.e. by Vannevar Bush and later Ted Nelson), but were not implemented the WWW.
According to these ideas, hyperlinks are bi-directional, which follows that a hyperlink is visible both in the document that features a link and also at the target document where the link is pointing to. The latter is a backlink. By this scheme, any document knows which other documents point to it.
The same concept is used by search engine algorithms, such as ‘page rank’ by Google, that function by creating an index of how many webpages link to a specific target website – or how many backlinks a given webpage has.
A link from this article to some other one, at the latter article will generate a backlink that would appear at the top right of the page (mentioned in this-article).
The root folder of the project contains the following files and folders.
This is a plain-text file, that can be edited with a general text editor.
article⁄Site Management 🧰
abstract⁄(Internal) Technical guide for site management.
keywords⁄internal, journal management
1. Introduction¶
This project runs on Sandpoints, a module/theme for the static website builder www⁄Hugo.
Content is organized with plain-text Markdown files. Collaboration/version control is managed via www⁄Git. The library is setup using www⁄Calibre and www⁄Accorder (for detailed instructions see article⁄Library management).
With the infrastructure in place, no programming is needed to maintain this project.
There are two main ways to manage and edit this site:
- On a browser via the preview site (only for content management; see §⁄management via online preview site).
- Via a local clone of the site repository (recommended, especially for major changes; see §⁄Site management via local repository clone).
Notes:
- Both workflows require write access to the project repository.
- For new content specifically see instructions at issue⁄Content Management.
2. Content management via online preview site¶
Note: for an extended guide for content management see issue⁄Content Management.
To access the online preview site append /_preview/ to the ⁄sitemap URL.
2.1. Preview site interface¶
The preview site enables a toolbar with five options (Screenshot 1), outlined in Table 1.
Screenshot 1: Interface of online preview site
| Preview site options |
|
| EDIT_THIS |
edit the current page |
| ADD_JOURNAL / ADD_ISSUE / ADD_ARTICLE |
create new entry by type |
| PUBLISH |
recompile the website and include all unpublished changes |
| ? |
open the repository website |
Table 1: Sandpoints online preview site edit options explained.
Note: Any action requires the user to be logged in to the project repository.
2.2. Edit existing entries via the preview site¶
All modifications (edits or new items) are carried out as Git commits (Screenshot 2).
To edit an existing entry navigate to the entry and click EDIT_THIS from the preview site interface.
After finishing all necessary edits, add a commit message, and press Commit changes.
Screenshot 2: Sandpoints preview site; interface for editing entries (EDIT_THIS).
2.3. Recompile and publish the site¶
To recompile the site and include any unpublished changes, either:
- include the keyword
!publish! in a commit message, or
- click the button
Publish of the preview website toolbar.
3. Site management via local repository clone¶
Making changes via a local clone of the site repository is recommended especially for larger scale edits, modifications of the site as well as for testing purposes. However, it has a slightly higher overhead and requires installing some software. The key benefits of making edits on a repository clone, over the preview page include:
- Running a localhost server for previewing changes (before they are committed).
- Making, and committing edits to multiple files concurrently (rather than editing entries individually).
- Having a local copy of all modifications before they are committed.
- Working with unpublished (draft) material.
- Making edits using a Markdown editor, rather than a Git textbox.
- Making non-destructive tests end edits.
- Making and testing theme modifications.
To manage the site you first need to install:
- www⁄Git – version control system.
- www⁄Hugo – a static website builder (§⁄see instructions below).
- It’s recommended to install the www⁄Go Language.
To manage the site you need the following tools:
- a Markdown editor (e.g. www⁄Obsidian), or a code editor (e.g. www⁄VS Code) or other general text editor to view and edit Markdown files. For Obsidian, open the
content folder as a vault.
- a command line interface (CLI) terminal. Windows users can use PowerShell or Git Bash (included with a Git installation). macOS users can use the default Terminal app, or www⁄iTerm2.
- a www⁄Git management tool. Experienced Git users can use a terminal. An easier workflow is to use a GUI Git software (e.g. www⁄Sourcetree), or the Git integration of www⁄VS Code.
3.2. Installing and upgrading Hugo¶
To install www⁄Hugo for your system see www⁄Hugo installation guide – Hugo is installed via a package manager, e.g. Homebrew for macOS or choco for Windows.
To upgrade Hugo:
- For Windows, if Hugo was installed with the
choco package manager, open a terminal as administrator and do:
choco upgrade hugo
- For macOS, if Hugo was installed with the
homebrew package manager, open a terminal and do:
brew upgrade hugo
To check the version of your Hugo installation, do:
hugo version
3.3. Cloning the site repository¶
Assuming you have access to the project repository, which should have a URL in the form of https://github.com/fake-repo/repo.git
Using a terminal, go to a folder where you want to clone the repository (e.g. c:/users/me/ for Windows), and do:
git clone https://github.com/fake-repo/repo.git
This will download a copy of all the material of the repository placed inside a folder with the name of the repository (in this example c:/users/me/repo).
3.4. Editing the site¶
To edit the site it’s recommended use a Markdown or other text editor such as www⁄Obsidian or www⁄VS Code. Editing only entails working with Markdown files within the /content/ folder of the project. See also the guides about content creation⦚article:int-contentnot found, article⁄naming and other conventions, and article⁄Markdown formatting.
3.5. Committing changes¶
If inexperienced with www⁄Git, use a GUI tool like www⁄Sourcetree, that can visualize commits, and branches of the repository, and display file version differences between commits.
3.6. Upgrading the Sandpoints theme¶
To upgrade the Sandpoints theme, navigate to the root folder of the project and run the following two commands, one at a time:
hugo mod get
hugo mod vendor
3.7. Building/compiling the online site¶
This site is automatically recompiled and updated by the server via Git commits. All commits that include the keyword !publish! will force the server to recompile the site taking into account the latest updates.
3.8. Building the site locally¶
Hugo has three main ways of compiling/building a site:
- Create a local real-time HTML server. This runs the site on a localhost server, that is accessible via browser on a URL like
http://localhost:1313/. This is useful for testing, for performing and viewing various edits locally, before changes are made public. This, especially because the localhost site is updated in real-time against changes made in its content. Also, because it allows previewing and editing unpublished (draft) content. See §⁄Create a local HTTP server.
- Compile a static HTML version of the site. The compiled site will be placed inside the directory
/public/. The contents of this directory can be uploaded to a server (e.g. via www⁄FileZilla). See §⁄Compile site locally via Hugo.
- Compile a portable and offline version of the site. This version of the site can be run and copied via a USB stick (for example). See §⁄Compile portable offline version.
3.8.1. Create a local HTTP server¶
Hugo allows running a localhost server which updates in real-time to follow any changes. To run a local server, navigate to the root folder of the cloned repository (by the previous example that would be cd c:/users/me/repo/), and do:
hugo server
To include all draft content in the localhost server, do:
hugo server -D
Hugo will start a local server (by default this is at http://localhost:1313/). The server will update with any change made to Markdown entries of the site, or to the theme.
Additional options for this command are:
--navigateToChanged -> automatically navigate to a page that is being edited.--noHTTPCache -> prevent HTTP caching.
3.8.2. Compile site locally via Hugo¶
Navigate to the root folder of the cloned repository via the terminal (by the previous example that would be cd c:/users/me/repo/), and do:
hugo
This will create (or overwrite) a directory named /public/ inside the root directory of the project.
The contents of /public/ can be uploaded to a file server (e.g. via FileZilla).
Note: To compile the site including draft content do: hugo -D (do not upload this version).
3.8.3. Compile portable offline version of the site¶
Hugo allows compiling portable versions of the site which can be copied and run on a USB stick (for example).
To render an offline portable version of the site, do:
hugo -e offline
This will also output the compiled site at /public/.
4. Debugging¶
For debugging see article⁄How to break or debug this site.
Note: If the site does not update after a publish command, there is likely an error that prevents the server from rebuilding the site.
article⁄Library Management 📚
abstract⁄(Internal) Library management guide for Sandpoints projects.
keywords⁄internal, library, journal management, guide
1. Introduction¶
Sandpoints is designed to be integrated with a library. The library is accessible via the red button at the top right corner of any content page, or at ⁄Library . Key features of the library include:
- library catalogue that can host various different file formats (e.g. ebooks, a/v media, and zip files; see §⁄populating the library);
- catalogue search by Author, Title, or Tags;
- presentation of library catalogue by book covers (default) or as list (select from top of the page);
- basic item metadata as well as the Librarian’s description of every entry.
- referencing library items in entries of this site via item id, and automating bibliographic citations.
2. Contribution workflow¶
For better library maintenance it is recommended to have single designated librarian.
The optimal way to contribute to the library is by submitting a bibliographic collection exported via Zotero (as described in §⁄Export bibliography collection). This can be easily integrated with the existing library by the managing librarian.
3. Required software for library management¶
- www⁄Calibre: free & open source eBook management software available for Windows, OSX, and Linux.
- www⁄Accorder: a Python command line tool – installation instructions below.
- A command line interface (CLI; such as Windows PowerShell, Git Bash, or Terminal or iTerm2 for macOS).
4. Library integration outline¶
- The library is maintained in www⁄Calibre.
- The library is excluded from the Git repository of the site, because of its sub-optimal way of handling large non-text files (e.g. PDFs).
- Adding items to the library can be done:
- directly to www⁄Calibre, or
- by loading a bibliography exported from www⁄Zotero via www⁄Accorder.
- To update the library, using the command line tool www⁄Accorder:
- Create a portable version of the library.
- Create a library catalogue (JSON dictionary located at
root/data/books/catalog.json)
- Upload the library via rsync.
5. Populating the library¶
Assuming you have a Calibre library running, you can add items from a range of file formats supported both by Calibre and Accorder. These are: pdf, epub, mobi, azw, azw3, cbz, chm, djvu, doc, docx, fb2, htmlz, mbp, md, txt, zip.
Note that a Calibre library entry can have multiple different files associated with it (for example both a pdf and an epub), however, it cannot have two files of the same format.
5.1. Via Calibre¶
- Click
Add books, or drag and drop a file (e.g. pdf, epub, zip) to Calibre.
- Select the item and click
Edit metadata, or press e.
- Add/correct the item title.
- Add/correct the name of the author(s). Author names are entered as in
Name Surname. Multiple authors are separated by ampersand as in Jane Doe & John Doe.
- Automatically download metadata. Calibre should be able to download metadata automatically from online databases (this works mostly for commercial books).
- With the author names and book title in place (ideally also the
ISBN – see step 7), . Click Download metadata or press d.
- Calibre will find available metadata sets, which might include book cover and item description or review. Check if the
- If Calibre fails to find available metadata, try to add the book ISBN and try again. Otherwise
- Add identifiers such as ISBN, or DOI in the field
Ids, as in isbn:12345674896. In case of multiple ids separate them with a semicolon(;).
- Add keywords separated by commas, as in
game design, monograph. Try to enter adequate keywords. If the item metadata were downloaded automatically review the keywords, as they are usually generic.
- Add publisher.
- Add publication date in the field
published.
- Make sure that the item has a book cover. If not either click
Download cover (which will try to find a cover from online databases), or click Generate cover to create one based on the item’s metadata.
- Click OK to save these changes.
5.2. Via Zotero¶
5.2.1. Export bibliography collection¶
To export items via www⁄Zotero do the following:
- Select an item, multiple items, or a collection, with valid file attachments which are stored locally (not on the Zotero cloud).
- Right click and select
Export Item(s).
- In the new dialogue select a BibTeX format and check the box
Export Files.
- Click OK, and select destination and filename (e.g.
mycollection.bib)
5.2.2. Import bibliography collection to Calibre (requires Accorder setup)¶
To import the Zotero collection to Calibre do (for setting up accorder profiles see next section):
accorder build -b path/to/zotero/export/mycollection/mycollection.bib profile-name
6. Building library standalone with Accorder¶
www⁄Accorder is a Python command line tool developed by Marcell Mars used for amateur librarianship.
It’s main features are:
- building a standalone portable web app version of a Calibre library accessible via
http,
- compiling a JSON catalogue of a standalone library,
- transferring bibliographic items within
.bib files (e.g. exported from www⁄Zotero and including files) into Calibre.
6.1. Install Accorder¶
To install accorder open a terminal and do:
pip3 install accorder
To upgrade accorder, do:
pip3 install --upgrade accorder
To check the version of your accorder installation do:
accorder --version
For a list and description of accorder commands do:
accorder --help
6.2. Accorder profiles¶
Accorder requires to setup a profile that is associated with:
- a librarian, and
- a local Calibre library directory.
To show existing accorder profiles, do:
accorder configuration --show
If an accorder profile was already setup, the previous command will output:
..PROFILE: profile-name
....LIBRARIAN: librarians-name
....CALIBRE LIBRARY: /path/to/calibre/library/
It will also show the path of the accorder.ini file with configuration profile information.
6.3. Create accorder configuration profile¶
To create a new accorder profile do:
accorder build -d /path/to/calibre/library/ -l librarians-name profile-name
This will setup a new profile and also build a standalone web app of the library.
The profile should now be visible in accorder configuration -s.
To edit elements of a profile configuration follow instructions in accorder configuration --help.
6.4. Build a portable library ¶
With accorder installed and a profile setup, to build a standalone web app of the library do:
accorder build profile-name
This will generate a portable version of the Calibre library by adding in the library directory:
- a
BROWSE_LIBRARY.html file, and
- a folder titled
static with necessary dependencies.
6.5. Build library catalogue¶
To build a library catalogue that allows referencing library items from Sandpoints do:
accorder build -j /path/to/export/catalog.json profile-name
This command will build the library catalogue in JSON format. The catalogue includes a unique ID and metadata fields for every item of the library.
The file should be named catalog.json, and should be placed inside a Sandpoints project in the directory root/data/books/catalog.json. Or, it can be directly compiled at that directory:
accorder build -j /path/to/root/data/books/catalog.json profile-name
Note: The catalogue file can be opened and viewed with any text editor. It is formatted in a single line. Some text editors feature tools that can auto-format JSON files so that they are more easily readable. For example, www⁄Notepad++ for Windows, features the plugin JSTools that can auto-format JSON.
7. Deploy library¶
To deploy a library use www⁄Accorder to:
-
§⁄build a library standalone web app.
-
§⁄build a library catalogue.
-
Place the library catalogue at the folder data/books/ of the project, as in:
/root/data/books/catalog.json
-
Build library standalone web app, and copy the entire contents of the Calibre library directory (including the static folder and BROWSE_LIBRARY.html) to …
For local projects the library can be copied to:
/root/content/library/
so that BROWSE_LIBRARY.html is at: root/content/library/BROWSE_LIBRARY.html
8. Link library item inside Sandpoints articles¶
To reference a library item inside a Markdown entry of this project do:
.
the-book-id is a placeholder for the unique id of a library item, which is in the style of 625e5562-38bc-4497-adaa-5142ef810c4a.
To find the unique ID of a library item, do either of the following:
- Open the library catalogue (
/root/data/books/catalog.json) with a text editor such as www⁄Notepad++. Search for the book that you want to reference, and copy its _id field.
- Open the ⁄Library find the item you want to reference, and copy the last part of it’s URL after
.../book/.
The following example links to the library item at: .../library/BROWSE_LIBRARY.html#/book/3ea44269-47eb-468b-906e-3104398aaa07
Thus, the item unique id is: 3ea44269-47eb-468b-906e-3104398aaa07.
Formatting:
Appearance:
Paul Coulton, Alan Hook, Petri Lankoski & Jussi Holopainen, 2017. bib⁄Games Design Research through Game Design Practice. Carnegie Mellon University: ETC Press.
Note: To overwrite the citation text that is automatically generated by the above syntax add some custom text in the square brackets.
article⁄Conventions 🧾
abstract⁄List of conventions to follow.
Leave draft: true
keywords⁄internal
DD.MM.YYYY
as in 13.05.1998.
2. File naming conventions¶
For file-naming conventions see Content creation instructions⦚article:int-contentnot found.
3. ‘int-’ prefix for files intended for Internal use¶
Files purposed to be for internal use, and are not public facing:
- start with
int- , like this file int-conventions.md
Their header file:
- features the keyword
internal
- the draft flag is set to true, as in
draft: true
- suffix the abstract with: Leave
draft: true
4. Images and attachments¶
Place images or other attachments related to a specific article in the appropriate subdirectory inside the root/static/ folder as in:
static/issue-x/author/image.png
So that it can be referenced from an article like so:
Note: When refencing an image or file provide a file path starting with a slash (/), and omit static.
article⁄How to break or debug this site 🐛 🧨⚫
abstract⁄Common reasons why the site breaks and would not compile, either locally or on the server.
keywords⁄draft, internal, guide, debug
1. Break local & online version¶
article⁄Features Requests / Issues 🦚
abstract⁄(Internal) List of feature requests.
keywords⁄internal, journal management, feature requests
1. Introduction¶
Here we can list feature requests and issues for Sandpoints, or for this project in general.
After being discussed, issues and requests should be communicated to and discussed with the Sandpoints developers, ideally by filing an issue on the www⁄Sandpoints repository.
To contribute an issue or request make a new subheading within §⁄feature requests, and add date and contributor. Resolved issues should be moved to the §⁄resolved section.
2. Feature requests¶
2.1. Expose item publication date (urgent)¶
Expose publication date of published material so that citation/reference scrapers can see when was something published.
My experimental www⁄Dotawo Zotero Translator cannot find any mention of a date.
- Issue posted on Sandpoints repository – CM 06.2023.
- Perhaps this can be done with the
.PublishDate www⁄page variable associated with the header field publishdate – CM 13.07.2023.
2.2. Add option to sort orphan entries in Sitemap by type¶
Currently, entries without parent (Print, Authors, Editors, etc.) appear in the ⁄sitemap sorted by creation date. It would be useful to have an option for sorting such orphan entries by type. For example, to have all Authors together, as well as Print entries and Editors appearing as grouped together.
- Issue posted on Sandpoints repository – CM 06.2023.
2.3. Associate print with target item¶
Currently, to print something, one has to go to the ⁄sitemap and look if there is a print entity for the item they are interested in.
It would be useful, as in backlinks, to be able to have a button such as ‘print this item’.
- Issue posted on Sandpoints repository – CM 06.2023.
Accorder seems to be optimized for book, and less for papers or chapters.
Necessary metadata fields seem to get lost in the transfer, like:
- ISBN of book chapter (ids)
- DOI of paper (ids)
- Conference and proceedings names
- URL of paper
- Book editors are
- For papers keep Conference and Proceedings name.
- Here is a Zotero item exported (file/annotation fields remove):
@incollection{graziano2019,
title = {Learning from \#{{Syllabus}}},
booktitle = {State Machines: Reflections and Actions at the Edge of Digital Citizenship, Finance, and Art},
author = {Graziano, Valeria and Mars, Marcell and Tomislav, Medak},
editor = {Colakides, Yiannis and Garrrett, Marc and Gloerich, Inte},
date = {2019},
publisher = {{Institute of Network Cultures}},
location = {{Amsterdam}},
url = {http://www.statemachines.eu/books/state-machines-reflections-and-actions-at-the-edge-of-digital-citizenship-finance-and-art/},
abstract = {some abstract},
isbn = {978-94-92302-33-5},
langid = {english},
keywords = {calibre,shadow libraries,syllabus}
}
- This in
catalog.json becomes (cover,formats, librarian, and id removed):
"xxxxxxxxxxxxxxxxxx": {
"title": "Learning from Syllabus",
"title_sort": "Learning from Syllabus",
"pubdate": "2019-07-15 00:00:00+00:00",
"last_modified": "2023-07-12 15:10:30.186344+00:00",
"tags": ["shadow libraries", "calibre", "syllabus"],
"abstract": "abstract",
"publisher": "Institute of Network Cultures",
"authors": ["Valeria Graziano", "Marcell Mars", "Medak Tomislav", "Yiannis Colakides", "Marc Garrrett", "Inte Gloerich"],
"identifiers": [{
"scheme": "bibhash",
"code": "nBlXgFSgppCE"
}
],
"languages": ["eng"]
}
3. Resolved¶
3.1. Add background to code blocks¶
Code blocks are useful not only for code but also for Markdown formatting instructions, or terminal commands having to do with site maintenance. I would be useful to be able to more easily distinguish code blocks like blockquotes. At the moment, it’s not easy to tell code blocks apart, or to distinguish them from text.
Edited CSS file (site.min.css) to:
Nevertheless:
- The actual background color, and padding should probably be reworked to fit better with the rest of the typesetting.
- Perhaps a 75% (or other) font size should be shared between
code and pre
Current changes are in commit f405856fbd – CM 11.07.2023.
3.2. Fix Zotero (.bib import) to Calibre¶
Accorder (see article⁄Library management), has a feature that allows reference collections exported from Zotero in .bib to be imported into the (Calibre) library. This doesn’t seem to work properly – CM 07.2023.
- We made it work via WSL terminal in Windows, using an older Calibre installation. In the meantime Marcell is working a new version of accorder in Go, which should arrive soon. For now library items can be forwarded via compressed
.bib to the librarian – CM 13.07.2023.
3.3. Remove date from keywords perspective¶
If you click on any keyword, the new page shows all entries with that keyword. It ALSO shows the date these were created (in my case file creation date is part of the MD header). It would be great if these dates could be omitted.
Resolved by editing out line 15 in ..theme/layouts/taxonomy/list.html
- Issue closed on 13.07.2022 – CM.
