Added Google Scholar links

Inspired by a paper with all it's citations linked to Google Scholar, I had another look at Google's service. It's useful if you can find the item you are looking for, and if the item is cross-referenced - Google figured out its citations, and other items that cited it.

I added Google Scholar links to the Gesamtausgabe App's (the GAApp?) Paper and Book details pages. In the case of papers, the URL linked to Google has the title (URL encoded; e.g., "Logic: The Question of Truth" => "Logic&3a+The+Question+of+Truth") and the Author's name in the form Google likes it (Initials and last name; e.g., "Andrew J. Mitchell" => "AJ+Mitchell"). I had to right a function to convert names to that format. With Books, if there's an author, the URL is the same as papers. If the book doesn't have an author, but has an editor, then URL uses the editor as the author. If the book has a translator but no author, then URL uses "M Heidegger" as the author.

The links appear to return pretty good results. Google Scholar either finds an exact match if its there, or reports it can't find it, without returning a bunch of unrelated results. Google should support searches with books' ISBNs t improve results when looking up books.

I'm considering also linking People in the GAApp to Google Scholar, and also individual Texts - most of the Texts in Wegmarken appear Google as individual citations.

I have not pushed this latest version of the GAApp to the cloud yet. It'll be in version 1.3.

[Update 4/16/15: I just discovered my simple author's name algorithm doesn't work with "Miguel de Beistegui" => "Md+Beistegui", no matches for his "The New Heidegger" on Google Scholar. The search has to be specifically for "M+de+Beistegui" to return the book. Other of his books require searching for specifically either "M+Beistegui" or "Md+Beistegui". Sheesh. Come on Google. Apply some intelligence, artificial or otherwise, and link all the variations to the same person, and have a query with any variation return all of a person's citations.]

Pernicious link rot

I started Ereignis, the web page in 1995, and one of the early gripes was that links stopped working, because the content was moved or deleted. When I was informed that a site had moved or I noticed a link was broken, I would update or remove it, but I did not regularly test the links and remove the dead links.

One of the goals with the Gesamtausgabe app is to only display valid links. Towards that goal, I've written a module that checks all the links in the database, and checks the links embedded in the paper and book page content. Once that was working, it was easy to re-purpose the code and point it at the the Ereignis pages on beyng.com, and have it check the links on Ereignis.

This I did. I skipped the Ereignis pages that have links by subject, and only repeat links that are already on the general pages. I included the book pages in the bibliography, where the links and mainly to authors, publishers, and reviews.

Out of 2691 hyperlinks on Ereignis, 705 were broken, 26%. Testing the 2691 links took 80 minutes. The oldest page of links, from the 1990's, had over 80% rotten links. 10% of the links from the last year have rotted. The distribution appears linear. Link rot occurs consistently. Surprisingly, links with the most rot were those to people, rather than papers. Links to institutional web sites are likelier to rot then links to individual web site. Universities and publishers are changing their web hosting software regularly and tossing their old content, while individuals are more likely to ensure that their URLs continue to work.

A helpful suggestion from History Today on broken hyperlinks.

Digital library researchers at Los Alamos National Laboratory found in a survey of three and a half million scholarly articles from scientific journals between 1997 and 2012 that one in five links provided in the footnotes suffered from 'reference rot'. Another survey, this time of law and policy publications, revealed that after six years nearly half of URLs cited had become inaccessible. Historians (perhaps unsurprisingly, given their profession) have been slower to place this most modern of problems at the top of their agenda. They are, however, not immune from its effect. An American study of two leading history journals found that in articles published seven years earlier, 38 percent of web citations were dead. Missing web pages can sometimes be relocated by academics through digital archives, the biggest of them being the Wayback Machine in San Francisco. A good many web pages, however, have not been archived and are permanently irretrievable.

A tool called Perma.cc was launched in beta phase in 2014. Developed by the Harvard Law School Library, it ‘allows users to create citation links that will never break’. If you want to secure the future of an Internet link in your footnotes, you create an archived version of the page you are referring to and anyone later clicking on your link will be taken through to the archived version. This ‘permalink’ does not repair Internet citations that have already decayed, but it does effectively fix the problem going forward. It has already been taken up by law reviews in America.

It would be cool if philosophy papers had links, instead of just referencing paper editions.

Azure Search and diacritics

I've been experimenting with Azure Search, to improve searching the Gesamtausgabe. I've got all the content indexed on Azure, and it's returning decent results; compensating for misspellings, and providing suggestions-as-you-type. I still need to figure out how to integrate with the Bootstrap typeahead control, before I update the website with the new search feature.

One of the features Azure Search doesn't have yet is asciifolding, so that a search for "αληθεια" will return documents containing "ἀλήθεια". Who can remember the polytonic Greek keyboard's diacritics' layout? And not every document uses diacritics consistently. If this feature is important to you, you can cast three votes for it here.

[Update March 9, 2015]
Asciifolding now works with Azure Search, with api-version=2015-02-28-Preview. The new release cadence from Microsoft is much better than the old days; "we'll fix that in the next Windows release". I've rebuilt the indexes and asciifolding is working in the app version that I'm currently working on. I hope to release it soon, several weeks.

In the fields you want to be searchable with asciifolding you set:

Analyzer = "standardasciifolding.lucene"

Drawing diagrams with SVG

I've figured out how to do large large curly brackets, like these in Thomas Sheehan's "Astonishing! Things Make Sense!" (P. 14):

I don't want to use bitmaps, like the picture above, because they don't scale, they're relatively large downloads, and their text isn't searchable. Today I figured out how to draw the curly brackets using SVG (Scalable Vector Graphics) to draw lines. A curly-bracket is two lines and four quarter circles.

Here's what it looks like in an HTML document. It won't render inside a blogspot blog, so you'll have to click there to see.

This is the code that renders the diagram.

    <svg height="700" width="800">
        <path d="M 200 260 a 7 7 0 0 1 7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 200 260 l 0 143" stroke-width="2" stroke="black">
        <path d="M 200 403 a 7 7 0 0 1 -7 7" fill="none" stroke-width="2" stroke="black">
        <path d="M 200 417 a 7 7 0 0 0 -7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 200 417 l 0 143" stroke-width="2" stroke="black">
        <path d="M 200 560 a 7 7 0 0 0 7 7" fill="none" stroke-width="2" stroke="black">

        <path d="M 420 160 a 7 7 0 0 1 7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 420 160 l 0 93" stroke-width="2" stroke="black">
        <path d="M 420 253 a 7 7 0 0 1 -7 7" fill="none" stroke-width="2" stroke="black">
        <path d="M 420 267 a 7 7 0 0 0 -7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 420 267 l 0 93" stroke-width="2" stroke="black">
        <path d="M 420 360 a 7 7 0 0 0 7 7" fill="none" stroke-width="2" stroke="black">

        <path d="M 560 95 a 7 7 0 0 1 7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 560 95 l 0 43" stroke-width="2" stroke="black">
        <path d="M 560 138 a 7 7 0 0 1 -7 7" fill="none" stroke-width="2" stroke="black">
        <path d="M 560 152 a 7 7 0 0 0 -7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 560 152 l 0 43" stroke-width="2" stroke="black">
        <path d="M 560 195 a 7 7 0 0 0 7 7" fill="none" stroke-width="2" stroke="black">

        <path d="M 560 315 a 7 7 0 0 1 7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 560 315 l 0 43" stroke-width="2" stroke="black">
        <path d="M 560 358 a 7 7 0 0 1 -7 7" fill="none" stroke-width="2" stroke="black">
        <path d="M 560 372 a 7 7 0 0 0 -7 -7" fill="none" stroke-width="2" stroke="black">
        <path d="M 560 372 l 0 43" stroke-width="2" stroke="black">
        <path d="M 560 415 a 7 7 0 0 0 7 7" fill="none" stroke-width="2" stroke="black">
        <g fill="black" font-size="14" font="Helvetica, sans-serif" stroke="none">
            <text x="70" y="390">EVERY λόγος =</text>
            <text x="70" y="410">σημαντιχός</text>
            <text x="70" y="430">A MEANINGFUL</text>
            <text x="70" y="450">UTTERANCE</text>

            <text x="220" y="240">λόγος ἀποφαντιχός</text>
            <text x="220" y="260">DECLARATIVE SENTENCE</text>
            <text x="220" y="280">I DECLARE P OF S</text>

            <text x="220" y="560">λόγος ἀναποφαντιχός</text>
            <text x="220" y="580">NON-DECLARATIVE SENTENCE</text>
            <text x="220" y="600">I WISH, HOPE, ASK, OR</text>
            <text x="220" y="620">COMMAND SOMETHING</text>

            <text x="450" y="140">AFFIRMATIVE</text>
            <text x="450" y="160">χατάφασις</text>

            <text x="450" y="360">NEGATIVE</text>
            <text x="450" y="380">ἀπόφασις</text>

            <text x="580" y="85">TRUE</text>
            <text x="580" y="105">ἀληθής</text>

            <text x="580" y="205">FALSE</text>
            <text x="580" y="225">ψευδής</text>

            <text x="580" y="305">TRUE</text>
            <text x="580" y="325">ἀληθής</text>

            <text x="580" y="425">FALSE</text>
            <text x="580" y="445">ψευδής</text>
        </g>
    </svg>

The problem with Google Scholar

Assume you want to find the most cited papers about a subject. There are library databases, that are professionally maintained, but they have a limited scope -- e.g., only cover a subset of all journals -- and it is difficult to get casual access to library databases. Then there is Google Scholar, which scans any paper Google can find, plus some databases, and is free to search. So Google Scholar should be a good source for citations.

But Google Scholar doesn't quite work.

For example, if you want to know how many times "Sein und Zeit" has been cited, you find:

"Sein und Zeit" is considered differently from "Sein und Zeit (1927)", "Sein und Zeit [Being and Time]", "sein und Zeit, tübingen", "Martin Heidegger: Sein und Zeit". Included too are any papers or books that include "Sein und Zeit" in their abstracts or titles, and onwards for dozens more pages of results.

The problem is that Google Scholar is just running automatically, trying to extrapolate citations from texts that are formatted in many different, inconsistent ways, and Google Scholar doesn't have editors who would realize that a set of different entries all refer to a single item and connect them. Algorithms don't understand the meaning of text. But they can get better.

How to get gesamt.html with PowerShell

How to download the file gesamt.html from Heidegger Gesamtausgabe app at gesam.azurewebsites.net. The app will generate a fresh copy from its database. The PowerShell commands follow:

# Sends a sign-in request by running the Invoke-WebRequest cmdlet. The command specifies a value of "fb" for the SessionVariable parameter, and saves the results in the $r variable. 
$r=Invoke-WebRequest https://gesam.azurewebsites.net/Account/Login -SessionVariable fb -UseBasicParsing

$rVerificationToken=($r.InputFields | Where { $_.name -eq "__RequestVerificationToken" }).value

# Gets the first form in the Forms property of the HTTP response object in the $r variable, and saves it in the $form variable. 
$form = $r.Forms[0]

# The next two commands populate the values of the "email" and "pass" keys of the hash table in the Fields property of the form. Of course, you can replace the email and password with values that you want to use.  
$form.Fields["Email"] = "nemo@nowhere.com"
$form.Fields["Password"] = "password"
$form.Fields["__RequestVerificationToken"] = $rVerificationToken

# The final command uses the Invoke-WebRequest cmdlet to sign in to the web service.
$r=Invoke-WebRequest -Uri "https://gesam.azurewebsites.net/Account/Login" -WebSession $fb -Method POST -Body $form.Fields

# The above is based on the Invoke-WebRequest help. Now get the gesamt.html into $r
$r=Invoke-WebRequest -Uri "https://gesam.azurewebsites.net/Band/Print" -WebSession $fb -Method POST -Body $form.Fields