Tuesday, October 13, 2020
New GA pages app
I've written a new Blazor app to complement GA App. It's GA Selis, for displaying a GA page.
I have over 28,000 GA pages, extracted from the best PDFs that circulate; under the beyng.com/tmp folder. Almost all of them were created from scanned book pages that were then OCR'd. I also have 1000+ GA pages which I've edited and corrected OCR errors, fixed the unlauts and Greek diacritics; under beyng.com/pages/de. GA Selis will look up a page by GA volume number and page number, first under pages/de and then under /tmp, so it will display an edited page if available, and if not, the raw OCR page, if that's found.
I also added a URL api, so it can be called to open a particular page; e.g.:
https://www.beyng.com/gaselis/?vol=79&pg=155
I added a "Greek help" feature. Double clicking on a Greek word will look up info on it and display it in a modal dialog. I copied the Greek data files I had created for the Greek words in the Volpi book/app and created a razor component with the Greek file fetching and dialog code. I also added a few lines of interop code to get the double-click event from JavaScript.
Thursday, December 26, 2019
New GA app
A few years ago I created a Gesamtausgabe app with the info about the volumes, translations, translators, related books, papers, etc.
The data was hosted in Azure, Microsoft's cloud, in a Sql Server database, and there was an IIS/MVC web site to generate web pages.
It is still running today.
Its functional problem is that the database and website go to sleep after 15 minutes of inactivity, and it takes a minute to restart the site, so, to look up a GA volume, it's quicker to use the simple text file version.
The old GA app required logging in with user IDs, in order to keep communications between the web site and browser browser, because of how SSL was implemented back then. Today, on beyng.com, SSL is now always available, anonymously; no logins or users IDs, authentication not required. Access to restricted documents is now controlled at the source -- e.g. access to Heidegger Circle proceedings is now controlled at heidegger-circle.org; if you are already logged there, links from GA app will open automatically -- so there is no need for the GA app to manage roles; different levels of authorization according to user ID.
I've written a new GA app. The new version does its processing in the browser instead of in the cloud. When it starts, it downloads all the data as text files to the browser, and the browser processes the data and generates web pages. The new version does not depend on anything running in the cloud, it only needs to download static files. I intend to keep updating this app, and make improvements.
The new GA app is built on a framework from Microsoft called Blazor, that runs on WebAssembly in the browser. WebAssembly is new and only works with current versions of browsers. The app will change as Blazor evolves.
The principal feature of the earlier cloud-based GA app that hasn't been reproduced in the new GA app is Search. The old cloud-based GA app use Azure for indexing and searching all the GA data, plus documents on beyng.com and files on other sites. Since search depends on the cloud, it is not implemented in the new GA app. I am looking for a way to search just the GA data in the client, and considering a new standalone search service that indexes all the files on beyng.com.
The old GA app required logging in with user IDs, in order to keep communications between the web site and browser browser, because of how SSL was implemented back then. Today, on beyng.com, SSL is now always available, anonymously; no logins or users IDs, authentication not required. Access to restricted documents is now controlled at the source -- e.g. access to Heidegger Circle proceedings is now controlled at heidegger-circle.org; if you are already logged there, links from GA app will open automatically -- so there is no need for the GA app to manage roles; different levels of authorization according to user ID.
I've written a new GA app. The new version does its processing in the browser instead of in the cloud. When it starts, it downloads all the data as text files to the browser, and the browser processes the data and generates web pages. The new version does not depend on anything running in the cloud, it only needs to download static files. I intend to keep updating this app, and make improvements.
The new GA app is built on a framework from Microsoft called Blazor, that runs on WebAssembly in the browser. WebAssembly is new and only works with current versions of browsers. The app will change as Blazor evolves.
The principal feature of the earlier cloud-based GA app that hasn't been reproduced in the new GA app is Search. The old cloud-based GA app use Azure for indexing and searching all the GA data, plus documents on beyng.com and files on other sites. Since search depends on the cloud, it is not implemented in the new GA app. I am looking for a way to search just the GA data in the client, and considering a new standalone search service that indexes all the files on beyng.com.
Monday, November 11, 2019
Turning texts into apps
I few years ago I wrote an app for the Volpi book, in Angular 5, to make it easier to study the text. For example, double-clicking on a Greek word popped up a screen about that Greek word.
This year I've been working two texts and trying to create apps for them. I started with the first ~200 pages of GA 19 (Plato's Sophist) and wrote a Blazor app. Blazor is an experimental framework for writing code in browsers using WebAssembly. The app hosts the German pages and English translation. I has some features like "hover over a Greek word to see glossary look-up". Some features, like responding to a double-click on a selected word, son't quite work in Blazor yet. I need to do some more experimenting with Blazor as it matures.
In the summer I joined a B&T reading group, and created a B&T app, with the first ~100 pages of that text. In addition to the German and English, this app also hosts Tom Sheehan's paraphrastic condensation; users can flip between English translation and paraphrase.
Creating apps for texts is labor intensive if the text is not ready -- e.g., needs OCR corrections.
This year I've been working two texts and trying to create apps for them. I started with the first ~200 pages of GA 19 (Plato's Sophist) and wrote a Blazor app. Blazor is an experimental framework for writing code in browsers using WebAssembly. The app hosts the German pages and English translation. I has some features like "hover over a Greek word to see glossary look-up". Some features, like responding to a double-click on a selected word, son't quite work in Blazor yet. I need to do some more experimenting with Blazor as it matures.
In the summer I joined a B&T reading group, and created a B&T app, with the first ~100 pages of that text. In addition to the German and English, this app also hosts Tom Sheehan's paraphrastic condensation; users can flip between English translation and paraphrase.
Creating apps for texts is labor intensive if the text is not ready -- e.g., needs OCR corrections.
Saturday, May 4, 2019
How to sort Greek in C#
Found here.
class GreekComparer : IComparer<string>
{
public int Compare(string s1, string s2)
{
return String.Compare(s1.Normalize(System.Text.NormalizationForm.FormD),
s2.Normalize(System.Text.NormalizationForm.FormD),
StringComparison.InvariantCultureIgnoreCase);
}
}
GreekComparer gc = new GreekComparer();
sortedWordList = wordList.Sort(gc);
Saturday, March 16, 2019
Normalize Unicode
When "Zoë" !== "Zoë". Or why you need to normalize Unicode strings
So important.
In javascript:
So important.
In javascript:
const normalized = str.normalize('NFC')Saturday, March 9, 2019
Make a Heidegger tool. Part I: assemble an archive
You're going to need Heidegger's texts. There are 100 volumes in his complete works. More than half of the German volumes are shared on the internet. A few are very good, all the characters in the electronic text are correct. A few are almost useless, the text can't be searched. Most of the usable texts are shared in PDF files.
1. Get the best version of each volume
Most PDFs have images of the pages and the text extracted (OCR) from the images. For our purposes, what matters is the quality of the text, not the quality of the images.
If you have a better OCR, extract better text from the images.
2. Convert the text to HTML files
Export the text from the PDF. Create an HTML page for each page of relevant text in the book. Try to get the most information possible from the PDF, like font (e.g. italics).
You will now have an archive of the best available texts. It'll be 90% reliable for simple words. 10% reliable for words with umlauts or Greek.
3. Correct the text
Update the text in the HTML pages to be correct, match what is on the printed page, in order to reliably search it. Most of the errors will be the results of OCR, which will make consistent errors, so you can make corrections across all the text files. Apply spellchecker; you'll need to add Heidegger's neologisms.
4. Put the HTML files on web server
5. Have search engines index pages
You can look up individual pages or search inside all texts.
The goal is to have 100% of the texts and that they be 100% correct in order to be able to search them reliably.
1. Get the best version of each volume
Most PDFs have images of the pages and the text extracted (OCR) from the images. For our purposes, what matters is the quality of the text, not the quality of the images.
If you have a better OCR, extract better text from the images.
2. Convert the text to HTML files
Export the text from the PDF. Create an HTML page for each page of relevant text in the book. Try to get the most information possible from the PDF, like font (e.g. italics).
You will now have an archive of the best available texts. It'll be 90% reliable for simple words. 10% reliable for words with umlauts or Greek.
3. Correct the text
Update the text in the HTML pages to be correct, match what is on the printed page, in order to reliably search it. Most of the errors will be the results of OCR, which will make consistent errors, so you can make corrections across all the text files. Apply spellchecker; you'll need to add Heidegger's neologisms.
4. Put the HTML files on web server
5. Have search engines index pages
You can look up individual pages or search inside all texts.
The goal is to have 100% of the texts and that they be 100% correct in order to be able to search them reliably.
Sunday, February 10, 2019
The Blazor Sofist
There's a new low level language in browsers called WebAssembly. Microsoft has built an experimental mechanism for running .NET virtual machines on WebAssembly called Blazor. That means that .NET languages like C# can now be used to write apps for web browsers.
I've written a simple Blazor app for a new, post-codex, "book". The first third of Heidegger's lectures on Plato's Sophist, which are about Aristotle's Metaphysics and Nicomachean Ethics. The app has the English and German text. I call the app Preliminary Sofist.
I've written the C# code to link Greek words on a page to their wiki entry, if that entry exists, the first time the word appears on a page. And, code to decorate Greek words with their English translation, if it appears in the glossary at the back of the book, so that the translation appears when the pointer hovers over the Greek word.
When Blazor's features improve, I want to add a dialog with Greek help, that pops up on double-clicking a Greek word, like I did with the Angular 5 Volpi app last year.
The app only works with Chrome. I use the IFrame srcdoc attribute to insert the page content into the app. Edge doesn't fully support HTML5.
I couldn't figure out how to get the app's URL routing to work from a sub-folder on a web site, so I had to host the app on its own domain.
I still have to proof-read and correct OCR errors from 2/3s of the German text, and add all the Greek words to the glossary and wiki links.
I've written a simple Blazor app for a new, post-codex, "book". The first third of Heidegger's lectures on Plato's Sophist, which are about Aristotle's Metaphysics and Nicomachean Ethics. The app has the English and German text. I call the app Preliminary Sofist.
I've written the C# code to link Greek words on a page to their wiki entry, if that entry exists, the first time the word appears on a page. And, code to decorate Greek words with their English translation, if it appears in the glossary at the back of the book, so that the translation appears when the pointer hovers over the Greek word.
When Blazor's features improve, I want to add a dialog with Greek help, that pops up on double-clicking a Greek word, like I did with the Angular 5 Volpi app last year.
I couldn't figure out how to get the app's URL routing to work from a sub-folder on a web site, so I had to host the app on its own domain.
I still have to proof-read and correct OCR errors from 2/3s of the German text, and add all the Greek words to the glossary and wiki links.
Tuesday, May 8, 2018
Why I haven't updated the GA App and its book links in a few years
What happened is that I was putting the books on my onedrive and then creating a link url on the site. The urls had a fixed format, with an id parameter. I store the id with the book record in the books table, and when generating the page HTML, generate the book's onedrive url with the id. A couple years ago, onedrive changed the format of their urls. The old book urls still work, but new urls have a different format. So I need to rewrite the code that generates the url. But I can't. I wrote the GA app four years ago, and Microsoft's changed its web tools (moved from old .NET to new .Net Core -- open source, works on Mac and Linux) and Visual Studio no longer supports the old .NET tools I used four years ago. So I need to upgrade to .NET core to compile any new code and rebuild the website. And I haven't got around to that upgrade project yet. Currently I'm more interested in learning client technologies, pushing more work to the browser, just serving files, and not having to keep a database running in the cloud to support the app.
Sunday, February 18, 2018
How to generate the Volpi book into a single file
Each page in the Volpi translation app is in a separate HTML file. I edit and update the individual files, and I don't maintain a single file with all of the book's pages.
However, you can generate a single HTML file with all the pages yourself, by pulling the individual pages from the app and concatenating them into a single file.
Here's how.
First create a start.html file with the HTML tags at the top:
Then create a end.html file with the HTML tags at the bottom:
Then use this PowerShell script to concatenate start.html, all the pages from the app, and end.html.
In between each page, the script inserts HTML with the page number. The padded page number is required for the page URLs - e.g. page 1 as 001. The UTF8.GetString stuff is required to keep the Greek characters from getting munged.
However, you can generate a single HTML file with all the pages yourself, by pulling the individual pages from the app and concatenating them into a single file.
Here's how.
First create a start.html file with the HTML tags at the top:
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Franco Volpi - Heidegger and Aristotle</title>
</head>
<body>
<h1>Franco Volpi - Heidegger and Aristotle</h1>
<h2>Translated by Pete Ferreira</h2>
<hr/><br/><br/>
Then create a end.html file with the HTML tags at the bottom:
</body>
</html>
Then use this PowerShell script to concatenate start.html, all the pages from the app, and end.html.
$bookContent = Get-Content 'start.html'
For ($pagenumber=1; $pagenumber -lt 118; $pagenumber++) {
$paddedpagenumber = ("{0:D3}" -f $pagenumber)
$url = "http://beyng.com/volpi/assets/EN/Volpi.$paddedpagenumber.html"
$resp = Invoke-WebRequest -URI $url
$bookContent += "<br/><br/><p style=""text-align:center"">$pagenumber</p>`n`r"
$bookContent += [system.Text.Encoding]::UTF8.GetString($resp.RawContentStream.ToArray()) } $bookContent += Get-Content 'end.html' $bookContent | Out-File VolpiBook.html
$bookContent += [system.Text.Encoding]::UTF8.GetString($resp.RawContentStream.ToArray()) } $bookContent += Get-Content 'end.html' $bookContent | Out-File VolpiBook.html
In between each page, the script inserts HTML with the page number. The padded page number is required for the page URLs - e.g. page 1 as 001. The UTF8.GetString stuff is required to keep the Greek characters from getting munged.
Wednesday, November 15, 2017
Volpi book
In April, after talking to some folks at Heidegger Circle 2017 in Walla Walla, I started translating Franco Volpi's Heidegger e Aristotele. I finished my first pass at the beginning of November. I now need to ensure I've translated words consistently, and that the English is readable. There are a few diagrams I need to render in SVG. But now that I have all the English text in HTML, I can create an app to review the pages.
I grabbed the Italian text from an e-book that circulates. I used a website to translate from the Kindle file to a PDF. Then I used the tools I've developed for other PDF books to get the text out into an HTML file per paragraph. The Kindle version of the book doesn't have page numbers - page 1 is the cover.
With previous book I extracted from PDF to HTML last year, I created an AngularJS app to display the book in browsers, making it easy page through or jump to any page. Earlier this year I wrote an app to try out Angular 2.0. Now, Angular is currently at version 5.0, so I decided to create the app for the Volpi book with the latest Angular and figure out how it works. I ran through the tutorial at angular.io last weekend, and today, I had a day off, and wrote an app for the book.
And it's fully functional: can navigate from ToC to any page, page through, jump to page, flip to the Italian text. And I got it running on beyng.com. Now I just need to pimp it up with some CSS. And add filters to allow users to select their favorite English word for controversial Heidegger terms. And finish the translation.
I grabbed the Italian text from an e-book that circulates. I used a website to translate from the Kindle file to a PDF. Then I used the tools I've developed for other PDF books to get the text out into an HTML file per paragraph. The Kindle version of the book doesn't have page numbers - page 1 is the cover.
With previous book I extracted from PDF to HTML last year, I created an AngularJS app to display the book in browsers, making it easy page through or jump to any page. Earlier this year I wrote an app to try out Angular 2.0. Now, Angular is currently at version 5.0, so I decided to create the app for the Volpi book with the latest Angular and figure out how it works. I ran through the tutorial at angular.io last weekend, and today, I had a day off, and wrote an app for the book.
And it's fully functional: can navigate from ToC to any page, page through, jump to page, flip to the Italian text. And I got it running on beyng.com. Now I just need to pimp it up with some CSS. And add filters to allow users to select their favorite English word for controversial Heidegger terms. And finish the translation.
Wednesday, March 1, 2017
Last year, after my last post, I decided what I really needed were true (correct) texts to work with.
I took the C# program I wrote to extract the text of SuZ from its PDF file, and modified the code to work with the PDF for another book. And then repeated the exercise, created another program for the next PDF book. The PDF of each book is sufficiently different from the others, that I haven't been able to write a general program for converting PDF pages to HTML pages. I wrote programs to covert the PDFs for Logic, Being & Truth, and Basic Concepts of Aristotelian Philosophy. I started from the highest quality PDFs, so that the generated HTML pages would not requires any editing. I modified the program I had written to upload the SuZ pages to the cloud and index them in the GA App's search system.
In the summer I went to a seminar with assigned Heidegger texts. I wrote programs for the texts' books' PDFs, to convert the PDF pages to HTML. Because the PDFs were low quality, I had to edit corrections on all the pages. I was learning AngularJS at the time and I wrote an angular app for each of the texts, to be able to easily find pages during the seminar. Links to the apps are in this post. I also added the pages to the GA App, to be able to search them.
At the Heidegger Circle in Chicago, I agreed with Andrew Mitchell to host a seminar on his book The Fourfold. When Andrew picked the Heidegger texts for the seminar I created programs to extract the text from the PDF files, and wrote angular apps to page through each text. For a couple of the essays, I also created HTML pages for their corresponding German pages from the GA, and linked the translation pages to the original pages. I created HTML pages for all the Bremen lectures, and Andrew's book, and indexed everything in the GA App.
I expect to keep generating more pages and adding them to the GA App's index.
I took the C# program I wrote to extract the text of SuZ from its PDF file, and modified the code to work with the PDF for another book. And then repeated the exercise, created another program for the next PDF book. The PDF of each book is sufficiently different from the others, that I haven't been able to write a general program for converting PDF pages to HTML pages. I wrote programs to covert the PDFs for Logic, Being & Truth, and Basic Concepts of Aristotelian Philosophy. I started from the highest quality PDFs, so that the generated HTML pages would not requires any editing. I modified the program I had written to upload the SuZ pages to the cloud and index them in the GA App's search system.
In the summer I went to a seminar with assigned Heidegger texts. I wrote programs for the texts' books' PDFs, to convert the PDF pages to HTML. Because the PDFs were low quality, I had to edit corrections on all the pages. I was learning AngularJS at the time and I wrote an angular app for each of the texts, to be able to easily find pages during the seminar. Links to the apps are in this post. I also added the pages to the GA App, to be able to search them.
At the Heidegger Circle in Chicago, I agreed with Andrew Mitchell to host a seminar on his book The Fourfold. When Andrew picked the Heidegger texts for the seminar I created programs to extract the text from the PDF files, and wrote angular apps to page through each text. For a couple of the essays, I also created HTML pages for their corresponding German pages from the GA, and linked the translation pages to the original pages. I created HTML pages for all the Bremen lectures, and Andrew's book, and indexed everything in the GA App.
I expect to keep generating more pages and adding them to the GA App's index.
Saturday, April 2, 2016
I think I'll be moving the GA App to dot net core this year, for the teaching-myself-current-tech side of the project. I built a dot net core web site for a small business last month, and that went pretty smoothly with Visual Studio.
Now I want to get more minimalist. This morning I figured out the fewest steps to getting a Hello World console app built and running, with dot net core RC2. And I'm sure I'll have to refer back to it.
Install nodejs to get ndm.
Now I want to get more minimalist. This morning I figured out the fewest steps to getting a Hello World console app built and running, with dot net core RC2. And I'm sure I'll have to refer back to it.
Install nodejs to get ndm.
npm install -g yo generator-aspnet
yo aspnet
Tell it to gen Console App HelloWorld
cd .\HelloWorld\
dnvm upgrade -r coreclr
dnu restore
dnx run
Wednesday, November 18, 2015
The Lurking Ligature
I ran into another horror on the road to making texts instantly available. In typography, a ligature is a special character made by joining two letters. For example, æ. There are also ligatures for prettifying text. There's an fl ligature, which won't render in Blogger, so I can't show it in this sentence. There's an Adobe PDF editor that includes ligatures as an option when you save a document. It will convert the fl's in the document into the special fl character, with the tops of the f and l connected, like the fl in 'reflections' from this book:
Yet another anomaly confusing machines, and people searching for 'reflections', although I expect Windows and Google handle ligatures automatically. Something to consider if you author PDF docs, and want your doc to be find-able.
Sunday, November 15, 2015
Added SundZ to GA App
Earlier this year I added full text search to the GA App, and I started adding content to the GA App. I started with some papers, and then added pages they cited, mainly individual German pages from GA volumes and pages from translated books. I would manually create each page in HTML and then manually add a hyperlink to the page, to the citation in the paper. Generally I would pick pages that were cited a lot, so adding a new page would allow me to add hyperlinks to multiple papers.
I would also add the new pages to the full text search index.
I created and added a couple hundred HTML pages, but creating each page manually didn't scale. I would take too long to create HTML pages for most of Heidegger's works. So at the beginning of summer I set that aside, and started looking into ways of generating the HTML pages automatically.
Most of the texts available on the internet are in PDF files. I spent some time experimenting with different tools and libraries of PDF functions. Getting the raw text out of a PDF file isn't difficult, but I wanted the HTML pages to look as much like the page in the book as possible. That meant getting information about the fonts (size, italics, etc.) and position (is a paragraph indented?).
I settled on using a library called pdfbox (it's on GitHub). I see they have a new version, 2.0, released last month. I'll have to try out the new one. I used the .NET version of 1.8.7. With pdfbox, I can get a list of all the characters on a page, with the coordinates of each, and font information.
I tried several different PDF files of books, and they were all sufficient different. That I ended up writing different programs to extract pages form different books. The programs have a common pattern and share common functions, but a single program that could handle all the books would be too complex. I decided to concentrate on the program to generate HTML pages from a PDF file of Sein und Zeit., SundZSteller.EXE. By October I had it working, generating HTML pages for all 437 pages of text in the book.
To date, when I need to add new data to the GA App, I rebuild the database; throw away the old database and create a new one. To add all 437 book pages, I wanted to figure out how to add a book with rebuilding everything. So I wrote a program that (1) adds the pages to Azure Storage, (2) adds the pages to Azure Search, and (3) adds the pages to the GA database. I finished the program yesterday and ran it and added the 437 pages to the GA App.
When someone searches for a term in the GA App, and Azure Search finds the term in a SundZ page, it returns a link to that page (and links to anywhere else the term was found). The link fetches the page's data from the database, displays the book page details web page with the data, and fetches the book page HTML from Azure Storage, and also displays the book page in the same web page.
Now I will go on to the next book, and write a program to extract all the pages from its PDF file. It should take less effort the second time.
I would also add the new pages to the full text search index.
I created and added a couple hundred HTML pages, but creating each page manually didn't scale. I would take too long to create HTML pages for most of Heidegger's works. So at the beginning of summer I set that aside, and started looking into ways of generating the HTML pages automatically.
Most of the texts available on the internet are in PDF files. I spent some time experimenting with different tools and libraries of PDF functions. Getting the raw text out of a PDF file isn't difficult, but I wanted the HTML pages to look as much like the page in the book as possible. That meant getting information about the fonts (size, italics, etc.) and position (is a paragraph indented?).
I settled on using a library called pdfbox (it's on GitHub). I see they have a new version, 2.0, released last month. I'll have to try out the new one. I used the .NET version of 1.8.7. With pdfbox, I can get a list of all the characters on a page, with the coordinates of each, and font information.
I tried several different PDF files of books, and they were all sufficient different. That I ended up writing different programs to extract pages form different books. The programs have a common pattern and share common functions, but a single program that could handle all the books would be too complex. I decided to concentrate on the program to generate HTML pages from a PDF file of Sein und Zeit., SundZSteller.EXE. By October I had it working, generating HTML pages for all 437 pages of text in the book.
To date, when I need to add new data to the GA App, I rebuild the database; throw away the old database and create a new one. To add all 437 book pages, I wanted to figure out how to add a book with rebuilding everything. So I wrote a program that (1) adds the pages to Azure Storage, (2) adds the pages to Azure Search, and (3) adds the pages to the GA database. I finished the program yesterday and ran it and added the 437 pages to the GA App.
When someone searches for a term in the GA App, and Azure Search finds the term in a SundZ page, it returns a link to that page (and links to anywhere else the term was found). The link fetches the page's data from the database, displays the book page details web page with the data, and fetches the book page HTML from Azure Storage, and also displays the book page in the same web page.
Now I will go on to the next book, and write a program to extract all the pages from its PDF file. It should take less effort the second time.
Wednesday, September 9, 2015
ἀλήϑεια != ἀλήθεια
Sad day today, to discover there are two thetas, the Greek letter θ, and the mathematical symbol ϑ; Unicodes U+03B8 and U+03D1 respectively. Some clever chaps, like Wikipedia and Google, are aware that both forms refer to the same character when embedded in words, and fold the mathematical symbol into the Greek letter. Sadly Adobe PDFs and Microsoft are clueless. If you search Bing for ἀλήϑεια, you get a bunch of papers by Thomas Sheehan, who, thankfully, appears to the only Heideggerian to use the mathematical symbol instead of the Greek letter. I discovered this looking up the definition of ἀλήθεια-1 in the PDF version of Sheehan's Making Sense of Heidegger.
I can fix this in the GA app, since all the texts I entered get their Greek normalized anyway -- all the ersatz Greek fonts from Win95 and Mac 9 days are converted to Unicode and non-standard diacritics fixed. The goal in the GA app is to make things easy to find, and not to resolutely follow the original text.
I can fix this in the GA app, since all the texts I entered get their Greek normalized anyway -- all the ersatz Greek fonts from Win95 and Mac 9 days are converted to Unicode and non-standard diacritics fixed. The goal in the GA app is to make things easy to find, and not to resolutely follow the original text.
Friday, July 3, 2015
character expansion or handling Eszett
This week I discovered that Azure Search does not handle character expansion, meaning that it considers Strasse and Straße to be different words. They are alternative spellings of the same word - it depends on what keyboard is ready-to-hand, or which official spelling directive rules your world. If you search Strasse you will not find documents containing Straße, and vice-versa. That's disappointing. I entered a feature request at the Azure Search site. Vote for this feature if you care!
In addition to using Azure Search to search all the documents, searching for a term in the GA App also searches the glossaries I've added to the database. There I've been able to enable character expansion. With search term Schluß, the glossary search returns Abschluss,
and searching for Schluss returns Schluß.
In the database, in table DeWords, with the records from a book (ISBN), find a record where the Word field matches the search query.
If there aren't exact matches in a glossary, GA App then looks for partial matches. InvariantCulture is automatic when searching for substrings, but always case sensitive, so:
In addition to using Azure Search to search all the documents, searching for a term in the GA App also searches the glossaries I've added to the database. There I've been able to enable character expansion. With search term Schluß, the glossary search returns Abschluss,
and searching for Schluss returns Schluß.
The trick to getting character expansion to work, is to tell the search that we want the search to be done with Invariant Culture; the default is Ordinal.
dw = db.DeWords.Where(w => w.ISBN == isbn).SingleOrDefault(w => w.Word.Equals(q, StringComparison.InvariantCultureIgnoreCase)); In the database, in table DeWords, with the records from a book (ISBN), find a record where the Word field matches the search query.
If there aren't exact matches in a glossary, GA App then looks for partial matches. InvariantCulture is automatic when searching for substrings, but always case sensitive, so:
dw = db.DeWords.Where(w => w.ISBN == isbn).FirstOrDefault(w => w.Word.ToLower().Contains(q.ToLower()));
Friday, June 5, 2015
Azure problems, sites down
My Azure sites are down until 6/19.
I have a Microsoft Developers Network subscription that gives me $150 in credits on Azure per month. I use Azure to host various experiments I have in the cloud (including my GA app and Ereignis beta), and pay for resources (web traffic, storage, etc.) from my Azure credits. Typically, my resource use is less than $50 a month.
Earlier this year, I signed up for the Azure Search beta. Search resources were free. Azure Search went live (became a billable product) last month, and there are now two tiers, free and standard. When Azure Search went live, Microsoft switched the users of the free beta, to the standard plan ($8/day), instead of the free plan. So Azure Search used up $8 every day, until it stopped working this morning.
My Azure web sites won't work again, until 6/18, when the monthly cycle rolls over, and they get another $150 again. At that point I'll shut down the Search service, create a new free search service, and re-index all the content for the search service.
I have a Microsoft Developers Network subscription that gives me $150 in credits on Azure per month. I use Azure to host various experiments I have in the cloud (including my GA app and Ereignis beta), and pay for resources (web traffic, storage, etc.) from my Azure credits. Typically, my resource use is less than $50 a month.
Earlier this year, I signed up for the Azure Search beta. Search resources were free. Azure Search went live (became a billable product) last month, and there are now two tiers, free and standard. When Azure Search went live, Microsoft switched the users of the free beta, to the standard plan ($8/day), instead of the free plan. So Azure Search used up $8 every day, until it stopped working this morning.
My Azure web sites won't work again, until 6/18, when the monthly cycle rolls over, and they get another $150 again. At that point I'll shut down the Search service, create a new free search service, and re-index all the content for the search service.
Tuesday, April 14, 2015
Added Google Scholar links
Inspired by a paper with all it's citations linked to Google Scholar, I had another look at Google's service. It's useful if you can find the item you are looking for, and if the item is cross-referenced - Google figured out its citations, and other items that cited it.
I added Google Scholar links to the Gesamtausgabe App's (the GAApp?) Paper and Book details pages. In the case of papers, the URL linked to Google has the title (URL encoded; e.g., "Logic: The Question of Truth" => "Logic&3a+The+Question+of+Truth") and the Author's name in the form Google likes it (Initials and last name; e.g., "Andrew J. Mitchell" => "AJ+Mitchell"). I had to right a function to convert names to that format. With Books, if there's an author, the URL is the same as papers. If the book doesn't have an author, but has an editor, then URL uses the editor as the author. If the book has a translator but no author, then URL uses "M Heidegger" as the author.
The links appear to return pretty good results. Google Scholar either finds an exact match if its there, or reports it can't find it, without returning a bunch of unrelated results. Google should support searches with books' ISBNs t improve results when looking up books.
I'm considering also linking People in the GAApp to Google Scholar, and also individual Texts - most of the Texts in Wegmarken appear Google as individual citations.
I have not pushed this latest version of the GAApp to the cloud yet. It'll be in version 1.3.
[Update 4/16/15: I just discovered my simple author's name algorithm doesn't work with "Miguel de Beistegui" => "Md+Beistegui", no matches for his "The New Heidegger" on Google Scholar. The search has to be specifically for "M+de+Beistegui" to return the book. Other of his books require searching for specifically either "M+Beistegui" or "Md+Beistegui". Sheesh. Come on Google. Apply some intelligence, artificial or otherwise, and link all the variations to the same person, and have a query with any variation return all of a person's citations.]
I added Google Scholar links to the Gesamtausgabe App's (the GAApp?) Paper and Book details pages. In the case of papers, the URL linked to Google has the title (URL encoded; e.g., "Logic: The Question of Truth" => "Logic&3a+The+Question+of+Truth") and the Author's name in the form Google likes it (Initials and last name; e.g., "Andrew J. Mitchell" => "AJ+Mitchell"). I had to right a function to convert names to that format. With Books, if there's an author, the URL is the same as papers. If the book doesn't have an author, but has an editor, then URL uses the editor as the author. If the book has a translator but no author, then URL uses "M Heidegger" as the author.
The links appear to return pretty good results. Google Scholar either finds an exact match if its there, or reports it can't find it, without returning a bunch of unrelated results. Google should support searches with books' ISBNs t improve results when looking up books.
I'm considering also linking People in the GAApp to Google Scholar, and also individual Texts - most of the Texts in Wegmarken appear Google as individual citations.
I have not pushed this latest version of the GAApp to the cloud yet. It'll be in version 1.3.
[Update 4/16/15: I just discovered my simple author's name algorithm doesn't work with "Miguel de Beistegui" => "Md+Beistegui", no matches for his "The New Heidegger" on Google Scholar. The search has to be specifically for "M+de+Beistegui" to return the book. Other of his books require searching for specifically either "M+Beistegui" or "Md+Beistegui". Sheesh. Come on Google. Apply some intelligence, artificial or otherwise, and link all the variations to the same person, and have a query with any variation return all of a person's citations.]
Friday, March 27, 2015
Pernicious link rot
I started Ereignis, the web page in 1995, and one of the early gripes was that links stopped working, because the content was moved or deleted. When I was informed that a site had moved or I noticed a link was broken, I would update or remove it, but I did not regularly test the links and remove the dead links.
One of the goals with the Gesamtausgabe app is to only display valid links. Towards that goal, I've written a module that checks all the links in the database, and checks the links embedded in the paper and book page content. Once that was working, it was easy to re-purpose the code and point it at the the Ereignis pages on beyng.com, and have it check the links on Ereignis.
This I did. I skipped the Ereignis pages that have links by subject, and only repeat links that are already on the general pages. I included the book pages in the bibliography, where the links and mainly to authors, publishers, and reviews.
Out of 2691 hyperlinks on Ereignis, 705 were broken, 26%. Testing the 2691 links took 80 minutes. The oldest page of links, from the 1990's, had over 80% rotten links. 10% of the links from the last year have rotted. The distribution appears linear. Link rot occurs consistently. Surprisingly, links with the most rot were those to people, rather than papers. Links to institutional web sites are likelier to rot then links to individual web site. Universities and publishers are changing their web hosting software regularly and tossing their old content, while individuals are more likely to ensure that their URLs continue to work.
One of the goals with the Gesamtausgabe app is to only display valid links. Towards that goal, I've written a module that checks all the links in the database, and checks the links embedded in the paper and book page content. Once that was working, it was easy to re-purpose the code and point it at the the Ereignis pages on beyng.com, and have it check the links on Ereignis.
This I did. I skipped the Ereignis pages that have links by subject, and only repeat links that are already on the general pages. I included the book pages in the bibliography, where the links and mainly to authors, publishers, and reviews.
Out of 2691 hyperlinks on Ereignis, 705 were broken, 26%. Testing the 2691 links took 80 minutes. The oldest page of links, from the 1990's, had over 80% rotten links. 10% of the links from the last year have rotted. The distribution appears linear. Link rot occurs consistently. Surprisingly, links with the most rot were those to people, rather than papers. Links to institutional web sites are likelier to rot then links to individual web site. Universities and publishers are changing their web hosting software regularly and tossing their old content, while individuals are more likely to ensure that their URLs continue to work.
Thursday, March 19, 2015
A helpful suggestion from History Today on broken hyperlinks.
Digital library researchers at Los Alamos National Laboratory found in a survey of three and a half million scholarly articles from scientific journals between 1997 and 2012 that one in five links provided in the footnotes suffered from 'reference rot'. Another survey, this time of law and policy publications, revealed that after six years nearly half of URLs cited had become inaccessible. Historians (perhaps unsurprisingly, given their profession) have been slower to place this most modern of problems at the top of their agenda. They are, however, not immune from its effect. An American study of two leading history journals found that in articles published seven years earlier, 38 percent of web citations were dead. Missing web pages can sometimes be relocated by academics through digital archives, the biggest of them being the Wayback Machine in San Francisco. A good many web pages, however, have not been archived and are permanently irretrievable.
A tool called Perma.cc was launched in beta phase in 2014. Developed by the Harvard Law School Library, it ‘allows users to create citation links that will never break’. If you want to secure the future of an Internet link in your footnotes, you create an archived version of the page you are referring to and anyone later clicking on your link will be taken through to the archived version. This ‘permalink’ does not repair Internet citations that have already decayed, but it does effectively fix the problem going forward. It has already been taken up by law reviews in America.It would be cool if philosophy papers had links, instead of just referencing paper editions.
Subscribe to:
Posts (Atom)