Something that is a recurring desire for me is an easier way to see what subject headings in LCSH have been changed. In particular, I’d like to have a way to get a list of those subjects that I can use to check to see if there are changes we need to make, particularly in our non-MARC systems like CONTENTdm and DSpace. Alma handles that pretty well for our MARC records, but for our non-MARC systems, I have to keep an eye on that and if we spot any subjects that have been updated, someone has to manually update those subjects in the metadata.

Right now, the only way I know of to see what subject headings have been changed by the Library of Congress is to look at the Monthly Approved Lists. (Although if anyone knows of a better, more programmatic way to get access to LCSH changes, please let me know!) The Approved Lists are helpful, but are not formatted in a way that is particularly easy to parse through. For this use case, in particular, they include a lot of subjects which are either new or where notes or reference fields have been updated but the term itself hasn’t changed. Even when the list contains mostly updated subjects, it isn’t easy to get a list of all of the terms that have changed that I can easily search for in our non-MARC metadata.

One of the use cases that I’m most interested in is when there is a list that includes a group of related subjects that are being updated together, like List 06 LCSH 2 (June 21, 2024) which includes a lot of updated subjects where “Racially mixed people” was changed to “Multiracial people.” I’d love to be able to get those updated across our metadata, and having a simple list of the old and new terms would be a big help.

So I wanted a way to parse the Approved Lists and get just a list of existing terms that have been changed. Fortunately, despite (or because of?) the fact that they are just plain text HTML pages, the monthly lists are pretty standardized, which made me think that they could be manipulated to get the data that I want out of them. There are probably more appropriate tools for this job, but the one that occurred to me first was OpenRefine, so that is what I went with. (Y’all know I love me some OpenRefine.) After some trial and error, I figured out a set of OpenRefine operations that can be reused to quickly and easily transform the text from a monthly list into a spreadsheet with all of the subjects on that list that were changed.

The JSON file with the OpenRefine operations is in my github here: https://github.com/elliotdwilliams/loc-monthly-list-changes

Here’s how to use it:

First, navigate to a monthly list that you are interested in, and simply use Ctrl-A and Ctrl-C to copy all of the text on that page. Then create a new project in OpenRefine. Use the “Clipboard” option to get data, and paste in the text you copied from the monthly list.

Data ready to be copied

And now that data pasted into OpenRefine

Here’s how the data will look when you first create the project in OpenRefine:

Next, copy the OpenRefine operation history and apply it to your project. (To do that, go to “Undo/Redo” in the upper left, select “Apply…”, and then paste in the JSON. You an also upload it as a file, but I usually just paste it in.)

And voila! Now you have a list of all of the subjects that have been changed, which can be exported as a CSV, spreadsheet, etc. It includes the subject term and MARC tag of both the old and new versions of the subject heading, as well as the identifier for the subject heading and a link to the term in id.loc.gov. (I can’t think of a scenario where the MARC tag would change between the old and new versions, but I included both just in case.)

I tried to comment each operation in the JSON file to provide a bit more context about what it is doing. The trickiest part was figuring out how to access the cell below a given cell in OR, but the cross function came to my rescue.

Currently, the operations only work for LCSH terms, not CYAC, LCDGT, etc. It might work partially for those other vocabularies, but the ID and link columns definitely won’t work. I’d like to play around with it some more to be able to parse what vocabulary each term is part of, but since I don’t work with those vocabularies regularly, it might not be a high priority.

I’d love to hear if this is helpful to anyone else, or if you know of more efficient ways to get this kind of data from LCSH!

Hello! I'm back with another edition of "Elliot writes out the process for how I did something, mostly for myself, but puts on the internet in case anyone else finds it useful." This one started out as one thing (counting percentages of MARC records in a set that contain a given field) and morphed into another thing, as well (use OpenRefine to add labels to LCC class numbers).

I've been discussing with one of my colleagues ways to enrich our MARC record for musical scores to improve their searchability in the catalog. One thing that we're interested in is including more 505 table of contents notes, particularly for collections or anthologies of printed music. There are a lot of reasons that would be nice, but a big one is that collections are one of the main sources in our collection for works by contemporary composers, composers of color, and women composers.

As we figure out if that's worthwhile and how we might approach it, I wanted to do some analysis of our notated music records to see how many have 505 notes. I wanted to break it down by LCC classification, since we're thinking that that might be a helpful way of identifying priority areas and chunking the project into manageable pieces. So here's what I did:

Getting the data

1. In Alma, did an advanced search for all physical notated music records, then exported those MARC records.
   • There were a total of 12,949 records, which is a lot but manageable for what I was hoping to do.
2. Used MarcEdit to export the fields I care about to a tab-delimited file: 001, 245, 050, 090, 505
   • Much later in the process, I realized it would have been helpful to include the 035 field at this stage. Whoops.

Processing in OpenRefine and class number cleanup

3. Imported the tab-delimited data into OpenRefine
4. Created a new column called "TOC?", where I entered either yes or no based on the presence of a 505 field in the record (using OR's "Facet by blank" facet)
5. Combined call number data from 050 and 090, and do some preliminary cleanup (mostly removing indicators/subfields and stray extra characters from the front)
   • At this point, I realized it would have been better to have the call numbers from the holdings records, rather than the bib records, but that would have been more complicated to get out of Alma so I decided to go with what I had. Good enough for this kind of exploratory analysis.
6. Used a regular expression to create a new column with just the first part of the call number, up to the first period that comes before a letter (hoping to get just the base classification number)
   • value.match(/[A-Z]+*\.?\d*).*/).toString()

Regex for getting the base class number

Getting more info about class numbers

7. Now I had the basic class number for each record. But I'm not super familiar with music class numbers, so I wanted to include the label for each number. I knew that on our old friend id.loc.gov, you can get data about each LC class in a variety of data formats, and you can construct the URL using the class number itself, e.g. https://id.loc.gov/authorities/classification/ML3551.html.  I first tried to work with the JSON versions, but I’m more comfortable with XML, so ended up using the RDF/XML format, e.g. https://id.loc.gov/authorities/classification/ML3551.rdf
8. Before making a bunch of URL requests to the Library of Congress, I used OpenRefine’s records mode and blank down feature, to make sure that I only had to make one request for each class number.  So only 757 URL requests, instead of 12,949.
   • I think I picked up this tip from a Programming Historian tutorial I was using to help figure out this process.

Records mode, always a lifesaver

9. Okay, now the good stuff: I created a new column in OpenRefine by fetching URLs based on Call No base column, using expression 'https://id.loc.gov/authorities/classification/'+value+'.rdf'.  Now I have a column full of XML!
10. Looking at the RDF/XML, I decided I wanted the <rdfs:label> field that contained the full chain of where this class number fits in the classification hierarchy.  The problem is, though, that there are multiple <rdfs:label> elements in the output – so using just plain parseXml().select(’rdfs|label’) wouldn’t work.  I want the instance of that element that is directly under the <lss:ClassNumber> element.  So I looked at the documentation for Jsoup (I know, what an idea, right?), and figured out I could use value.parseXml().select("rdf|RDF > lcc|ClassNumber > rdfs|label")[0].toString() to get exactly the element I wanted.
    • (This makes it sound a lot more straight-forward than it actually was. Imagine lots of flailing around and frantic googling at this stage, particularly when I was trying to use JSON results before switching to XML.)
    • Pro-tip that I learned in this process: if your XML elements have namespaces, you have to enter them in the select expression as "rdfs|label", instead of "rdfs:label".

Class number, retrieved XML, and the full hierarchy label for that number

11. Ta-dah!  Now I have a column with the full LCC description for each class number in my dataset.  I used fill down in OpenRefine to add the class number and label to all of the rows.
12. There were about 7 records where the process didn’t work because the data wasn’t structured normally or the call number was missing from the bib, so I added those manually.

At this point, I thought I might be fancy and do something with pandas (which I'm very much a beginner with) to count the number of records without a 505 per class number. But I decided I knew how to do what I wanted to in Excel, so I just did that instead. 🤷 I used some simple deduping and COUNTIF formulas to get the number of records per class number, and the number in each class number that do not have a 505 field. And it includes the full classification hierarchy label for each number, which is helpful. That allowed me to highlight all of the rows with "Collection" in the classification label, which has already been helpful in pinpointing some class numbers to explore in more depth.

The final product in Excel

I’ve been thinking about subject headings a lot lately (as one does), and it inspired me to return to some work that I had started doing while I was at TDL, thinking about subject heading uniqueness in DPLA.

One of the things that I’ve noticed in my own metadata work, and that was confirmed for me when I was looking at other people’s metadata as part of my TDL job, is that subject headings for digital collections are often very specific and detailed. On top of that, there aren’t good rules for applying subject headings to digital collections items (in the way that there are for MARC cataloging), and the people who create digital collections metadata often don’t have a lot of expertise in LCSH structure and application rules (no shade, it’s a very niche and abstruse area to have expertise in). All of that combined means that I have a strong suspicion that a lot of subject terms used in a given institution’s digital collections are unique to that institution.

That’s all well and good within that institution’s repository, but when you start sharing metadata with an aggregator like DPLA, I think that starts to become more of a problem. If we assume that one of the purposes of applying subject headings is to link similar items together, then having subject headings that are unique to an institution don’t serve that purpose, or the purpose of aggregation, very well.

So I wanted to find a way to easily identify what subject headings for a given institution are unique in DPLA. I was inspired a lot by the research that Mark Phillips and Hannah Tarver have published around metadata record graphs, also looking at subject headings in DPLA. Their research is a lot more detailed and uses network analysis to explore how subject headings link records together within a large corpus. My goal was more modest: as a metadata creator, I want to see what subject headings in my institution are unique, to potentially inform metadata creation practice at my library.

As I do in most situations, I turned to my faithful friends: python and the DPLA API. (Can we talk for a minute about the DPLA API, and how simple and easy to use and well-documented it is? It’s really great.) I ran through a couple of iterations of the script before I settled on a method that I like. Here’s the script I came up with: https://github.com/elliotdwilliams/dpla-subject-search/blob/main/dpla-uniq-subjects.py (In that Github repository, there are a few other variations that do parts of that work, if you’re looking for something other than strictly unique subject terms.)

Basically, the script takes a given contributing institution in DPLA, searches for that institution, and grabs a list of the most common subject terms in that institution’s contributed records. (You can get up to 2,000 subject terms, because that is the limit that the API will return as a facet.) Then, it searches for each of those subject terms in DPLA, and checks to see if it is used by only one institution. If it is, that subject term is written to an output file. It also generates a percentage for how many of the subject terms it searched are unique.

I ran it today for UTSA Libraries, and of our top 1000 subject headings in DPLA, 323 are unique to us. Which, honestly, I feel like a 32% uniqueness percentage is pretty good! Looking at the list, a lot of them are unique because they are about San Antonio, which makes sense (either local places like “West Side (San Antonio, Tex.)” or pre-coordinated strings like “Lesbians--Texas--San Antonio”). There are also quite a few that show up as unique because of some idiosyncratic formatting choices that have been made in the past. There are some headings, though, which I wouldn’t have expected to be unique in DPLA, like “Immigrants, English” - offhand, that looks to me like a correctly formatted LCSH term, but I’m definitely going to double-check. I also spotted a couple of typos, which now that I know about them I can fix them!

I wouldn’t call it a groundbreaking metadata analysis tool by any means, but I’m glad to have this script available for exploring subject terms in DPLA.

It's been a minute since I wrote anything on this here blog, but I was working through a process today that I wanted to document and thought that other folks might be interested in.

This week, a group of coworkers and I were discussing whether there is an LCSH for "Cuban diaspora" (there is not), and whether there should be (we think that might be useful, and might eventually want to work on a proposal). There are a handful of other LCSH terms for "[blank] diaspora", and I wanted to be able to view them all at once, rather than paging through them individually in ClassWeb. That way I could compare the See From and See Also From references, the public notes, and the works cataloged, to get a sense of what the format and structure of this group of subject headings is. So basically, what I wanted to do was to download a batch of LCSH records in a format that I could open in OpenRefine to examine.

LCSH records are available in a variety of formats from id.loc.gov, so I know the records are there on the web. The records are available in MARCXML, JSON, RDF, and a number of other flavors. I just needed a way to identify the ones I wanted and download them in batch.

Here's what I ended up doing. As always, I'm sure there are other, probably faster or simpler ways to do this, but this process worked for what I wanted and used tools that I already knew how to use:

1. Search LCSH for "diaspora" in id.loc.gov
2. Literally copy and paste from the search results into a spreadsheet. There were 55 total, so it was only 3 pages of results that I needed to copy.

LSCH search results, ready to be copied

3. In the spreadsheet, remove the subjects I'm not interested in, isolate the identifiers (e.g. sh2006004206), and save the list of identifiers as a text file. Now I have a list of all of the identifiers for the relevant subject headings.
4. Now the part that took a bit more figuring out: Use curl to download the files in bulk. Here's the command I used:

in $(cat Desktop/DiasporaSH_ids.txt); do curl -o “Desktop/diaspora/$ID.xml” “https://id.loc.gov/authorities.subjects/$ID.marcxml.xml”; done

Basically, what that does is read the list of identifiers, copies them into a variable ($ID), then runs a curl command using that variable to build the URL and saves the output, in this case as an XML file. I use GitBash as my shell for running commands like this. As usual, it took me some time to figure out how to get the "for" loop to work (it always takes me a while to remember how to structure it properly). And because I use a windows computer, I also had to remove the carriage return characters at the end of the DiasporaSH_ids.txt file, so I’d stop getting an “Illegal characters found in URL” message.  I used dos2unix to do that.

5. Now I have a folder full of MARCXML files!
6. Repeat the curl command for RDF and JSON files, or other formats as needed.

For what I wanted to do today, the MARCXML turned out to be the most useful. I went through a whole process using MarcEdit to convert them into .MRC files and then export the records to tab-delimited text, which I then loaded into OpenRefine. I'm sure another tool could have done that more easily (probably pymarc?), but I knew I could get it done with MarcEdit, and just went with that.

And voila, I had what I wanted!

As a way to start thinking more about that and starting to wrap my head around it, I spent some time pulling together a list of all of the RDA rules that deal with religious works or religious bodies.  I used the current version of the RDA Toolkit (not the Beta Toolkit) and did searches for “religion”, “religious”, and “church.” I decided to only include rules if the instructions are specific to a religious body or text.  I didn’t include if a religious body or text was used as an example, but the rule does not specify to treat it differently (although that would be an interesting addendum). I also didn’t include rules that only include a reference to another rule that is specific to religious bodies or texts.

The list is currently stored in a Google Doc. I won’t guarantee that this list is comprehensive.  I may have missed some rules! Please let me know if you can think of any that should be included, or any other search terms that might be interesting.

I have a few ideas of what to do with this next, but even just this list of rules is interesting in and of itself.  It helps me get a sense of when and why religion appears in RDA, as well as what religions and faith traditions are most explicitly discussed.

'People taking a gondola ride along the Coral Gables Waterway. Coral Gables, Florida' "Recognizing the importance of having a luxury resort hotel in the city, George Merrick turned to John McEntee Bowman, President o...'https://t.co/27MqoSFJtU

I love twitter bots. There are some really cool ones doing interesting things with cultural heritage materials:

• @NYPLEmoji, which responds to emojis with matching items from NYPL’s collections
• @pokemon_nypl, which tweets collections images with Pokemon added to them
• @PDcutup, which creates collages out of two public domain images from two different institutions based on similarity in their titles
• @leia_quotes, which tweets all of Princess Leia’s lines of dialogue (not cultural heritage related, but I still love it)

I’ve been slowly working my way through the Codecademy Python course, and I thought that a twitter bot would be a good chance to practice my Python coding skills. At work, I’ve been working a lot with a bunch of other amazing people on getting the Sunshine State Digital Network (SSDN) established as the DPLA hub for the state of Florida, and as part of that, I’ve had the chance to play around with the DPLA API. So, combining all of those things, I decided to make a twitter bot that randomly tweets out items that SSDN has contributed to DPLA. It seemed like a good project for a variety of reasons: it was doable, there are lots of tutorials on creating a twitter bot using Python, and it combined several things I’ve been interested in lately.  And so, @SSDNbot was born!

This was my first real project in Python - I’ve tinkered around with other people’s scripts, and done some tutorials, but this was my first start-to-finish project. That made it a little intimidating, but like I said, there are plenty of Python twitter bot tutorials out there. Some of the ones that I found most helpful are:

• http://briancaffey.github.io/2016/04/05/twitter-bot-tutorial.html
• https://jitp.commons.gc.cuny.edu/make-a-twitter-bot-in-python-iterative-code-examples/
• https://github.com/tommeagher/heroku_ebooks
• https://github.com/llamafarmer/bitcoin_tweeter (just ignore the fact that it's about bitcoin…)

I could maybe have followed some of these tutorials without really knowing what I was doing, but having been working through the Python course on Codecademy really helped. I also used two other DPLA-related bots as models, which were super helpful (@DPLAbot and @dplafy [links go to their GitHub repositories]).

Reading tutorials like those above also introduced me to Heroku, which I wasn’t at all familiar with. Basically, Heroku provides isolated Unix machines that you can use to run scripts (I’m sure it does much more, but that is how I’m using it). And the best part is that there is a free version! So I launched my script on Heroku, and was able to use the Heroku scheduler to run it every 12 hours. Which is great, but I could not for the life of me figure out how to write the scheduling into the script itself (which was extraordinarily frustrating).

The Python script harvests 500 items at a time from DPLA (because that is DPLA’s API limit), limited to items contributed by SSDN, then randomly picks one of those items to tweet. It composes a tweet that includes the item’s title, the item’s description field (if it has one), and a link to the item in DPLA. Initially, I was just using the title, but so many items have titles that aren’t really very descriptive, so I decided to include the description. Figuring out how to write the logic to only include the description field if it exists was tricky, but was a good thing to learn how to do. I also ended up having to rewrite the script in Python 3, instead of Python 2.7, because of the different ways Unicode is handled in the two versions. Character encoding problems are just inescapable, apparently.

One of the things that this project reinforced for me is the number of things you need to know to get a technical project like this going. (I actually wrote about a similar thing, way back in the first entry on this blog, back when I was just a wee iSchool student.) I had most of the Python knowledge that I needed (or could pick it up along the way), but there’s always way more to it than that. I had to learn the slightly different way you use pip on Python 2.7 vs. Python 3. I had to learn about and understand how to deploy something on Heroku. I had to get a better understanding of how to actually use GitHub. There are always more pieces to master than I expect, and I often don’t know I need to know them until I’m confronted with them. I don’t know that there is an easy solution to this, but I think it’s important to remember, especially when thinking about how to create usable documentation and tutorials.

'Jones, Randolph' 'A record describing Jones, Randolph, a colored voter from Leon County, who registered to vote in 1867.'https://t.co/btiuXMEN6h

— SSDNbot☀🌴🐊 (@SSDNbot) June 29, 2018

I’ve really enjoyed watching my little bot tweet away. Seeing individual items has helped me notice things about our collections that I wasn’t aware of, and putting them in my twitter feed has helped me contextualize them in new and interesting ways. I also think there is real value for metadata librarians to build things like this. For example, I’ve become more aware of how important distinctive titles are after seeing how unhelpful some titles were on their own.

So what’s next for SSDNbot? A few weeks after the bot started tweeting, I noticed that some of the links to DPLA weren’t working - they would sometimes (but not always) lead to a 404 page. I’m still not sure what was going on there, so I’m keeping my eye on that. There are a few improvements I’d like to make to the bot, as well. It would be really cool if it could include the actual image from the item, instead of just the link. That is complicated by two factors: the images aren’t stored by DPLA, so it would need to pull them from the system of origin; and since the script runs on Heroku, it doesn’t really have a place to store the image files before tweeting them. Possibly not insurmountable obstacles, but will take some thinking. I’d also really like to find a way to pull from all 148,000 items contributed by SSDN, instead of just the first 500. I’m not sure what the way to do that without clobbering the DPLA API might be, though. I’ve also thought about doing some sillier things with it - replacing text in the metadata with emojis, for example. Who knows; the twitter sky is the limit!

I thought that other people might have a similar use case for this workflow, and wanted to make sure and save it for myself, so I decided to write up some quick documentation here.  I’m fairly confident that there are other, possibly much simpler, ways to do this, but this was a method that I could do using tools that I already (mostly) was familiar with.

First, export the metadata for the collection from CONTENTdm, and open it in Excel. Then copy two columns into a new spreadsheet:

• Column A should contain containing the identifiers, which will become the filenames for the text files (for us, these are our “Digital IDs”, which match the filenames of the corresponding TIFFs)
• Column B should be the full text transcription (for us, this is simply called “Full Text”

Delete any rows without text in the Full Text column, using “Go To Special”

To save each row as its own text file, I used a quick macro that I adapted from responses to a Stack Overflow question. The macro saves each cell in Column B as a txt file, with the value of Column A as the filename:

Sub savemyrowsastext() Dim saveText As String   For Each cell In Sheet1.Range("A1:A" & Sheet1.UsedRange.Rows.Count)      saveText = cell.Text      Open "C:\Users\ewilliams\Desktop\" & saveText & ".txt" For Output As #1      Print #1, cell.Offset(0, 1).Text      Close #1 Next cell MsgBox ("Done") End Sub

Ta-dah!  Now you have text files for each page/image in CONTENTdm!

In looking through the text files, though, I realized that the line breaks in the transcriptions got lost in the process.  (I suspect CONTENTdm doesn’t output the line breaks in the metadata at all, which makes sense.)  I wanted those line breaks, though, since they were present in the transcriptions in CONTENTdm, and help make the transcriptions more user-friendly.

Looking at the files, I discovered that in the exported metadata, what showed up in the CONTENTdm front-end as two line breaks was turned into four blank spaces.  I can’t guarantee that is how CONTENTdm will always export that, but it was in my case.

Well, that should be easy enough to fix en masse using the command line.  After googling around and testing various ways of manipulating text files on the command line, I settled on using the sed command. (Various attempts to use the tr command or perl didn’t work, for reasons I wasn’t willing to put the time into figuring out.) I did a little more research/refreshing on how to encode newline characters, and ended up with this:

sed –i ‘s/    /\r\n\r\n/g’ *.txt

It’s worth noting that sed only works on a bash shell (I use Git Bash).  It’s also a good idea to make backups of the text files before running this, just in case something goes wrong (although you could always just re-export them from the Excel file).

So that’s it.  Relatively straight-forward, although as I mentioned, I’m sure there are quicker/cleaner ways to do it.  I’d love to hear any feedback or suggestions for how to improve this process!

[Updated June 2018: Originally, this post had an embedded Storify with all of the tweets.  Now that Storify is no more, you have to go to Twitter to read the thread.]

Thread: https://twitter.com/elliot_dw/status/900115599019343873

#archivalfeelings, a thread about metadata and history and connection:

— Elliot D. Williams (@elliot_dw) August 22, 2017

And the update with the digital collection link: https://twitter.com/elliot_dw/status/903354735452880896

The slides of Félix González-Torres' work that I was talking about in this thread are online now: https://t.co/9KDTerIpjA https://t.co/VBmEUnBEK5

— Elliot D. Williams (@elliot_dw) August 31, 2017

The timeline below is my first, very imperfect attempt at mapping out the big markers in the history of MARC.  A couple of caveats: it is very much the result of my interests and my knowledge (other people would likely highlight other milestones); it is very US-centric (given my familiarity with and interest in MARC21 and USMARC); it is incomplete (I know it is missing things, and hopefully I will add to it over time). I also don’t love this timeline format, but I decided to use this simple but unlovely Wordpress plugin until I find something better.

I’d love feedback, suggestions, criticism, etc.  Tell me what I’ve forgotten, point me to other sources, correct me on things I got wrong!  But in the meantime, please enjoy the history of our goofy but lovable data format.

Color code: blue is general policy and history; green is bibliographic format; red is all other formats; yellow is format integration.

[svtimeline]

[svt-event title=”CLR report on automation at Library of Congress” date=”1963” class=”svt-cd-blue” ] [/svt-event]

[svt-event title=”MARC Pilot Project (MARC I)” date=”1965-1968” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”MARC II introduced at ALA” date=”1967-07” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”LC releases specification for MARC II” date=”1967-08” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”MARC II operational” date=”1969-03” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”Serials and maps formats released” date=”1970” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”Film format released” date=”1971” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”Original manuscript and music formats released” date=”1973” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”MARBI takes on role of MARC Advisory Committee” date=”1973” class=”svt-cd-blue” ] [/svt-event]

[svt-event title=”MARC Authorities Format published” date=”1976” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”Review based on publication of AACR2” date=”1978-1979” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”Bibliographic formats published together as MARC Format for Bibliographic Data (MFBD)” date=”1980” class=”svt-cd-yellow” ] Each field had “validity designators” for different forms of material. [/svt-event]

[svt-event title=”The USMARC Principles: The Underlying Principles published” date=”1982” class=”svt-cd-blue” ] [/svt-event]

[svt-event title=”Format integration broadly undertaken” date=”1983-1988” class=”svt-cd-yellow” ] [/svt-event]

[svt-event title=”USMARC Format for Holdings Data published” date=”1984” class=”svt-cd-red” ] First edition [/svt-event]

[svt-event title=”Format Integration guiding principles published at MARBI Midwinter meeting” date=”1984” class=”svt-cd-yellow” ] [/svt-event]

[svt-event title=”New edition of Authorities format published” date=”1987” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”Discussion Paper 16: Format Integration Considerations released and discussed” date=”1987 (Spring/Summer)” class=”svt-cd-yellow” ] [/svt-event]

[svt-event title=”Proposal 88-1 (Format Integration) voted on and approved, with amendments, by USMARC Advisory Committee and later by LC” date=”1988 (Late Summer)” class=”svt-cd-yellow” ] [/svt-event]

[svt-event title=”USMARC Format for Bibliographic Data published, incorporating format integration changes” date=”1988” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”New version of USMARC Format for Holdings Data published” date=”1989” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”Provisional format for community information approved by MARC Advisory Committee” date=”1992” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”New edition of USMARC Format for Authority Data published” date=”1993” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”USMARC Format for Community Information published (first edition)” date=”1993” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”New edition of USMARC Format for Bibliographic Data published” date=”1994” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”USMARC and CAN/MARC communities work on integration” date=”1994-1997” class=”svt-cd-blue” ] [/svt-event]

[svt-event title=”Update No. 2 to 1994 edition of authority format published, bringing USMARC and CAN/MARC into alignment” date=”1997-03” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”Update No. 3 to 1994 edition bibliographic format published, bringing USMARC and CAN/MARC into alignment” date=”1997-07” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”MARC 21 published” date=”1999” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”MARC 21 Format for Authority Data published” date=”1999” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”MARC 21 Lite Bibliographic Format published” date=”1999-02” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”MARC 21 Formats for Authority Data, Holdings Data, and Community Information published” date=”2000” class=”svt-cd-red” ] [/svt-event]

[svt-event title=”MARC 21 XML released” date=”2002-05” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”British Library adopts MARC 21 to replace UKMARC” date=”2004-06” class=”svt-cd-blue” ] [/svt-event]

[svt-event title=”New edition of MARC 21 LITE Bibliographic Format published” date=”2006” class=”svt-cd-green” ] [/svt-event]

[svt-event title=”RDA/MARC Working Group formed” date=”2008” class=”svt-cd-blue” ] [/svt-event]

[/svtimeline]

Sources

• Format Integration and its Effect on the USMARC Bibliographic Format, 1992 ed. (Washington: Library of Congress)
• Library of Congress. MARC21 format specifications.
• McCallum, Sally H. (1990). "Format Integration : Handling the Additions and Subtractions." Information Technology and Libraries, 9(2), 155–161.
• Seikel, Michele, & Steele, Thomas (2011). "How MARC Has Changed: The History of the Format and Its Forthcoming Relationship to RDA." Technical Services Quarterly, 28(3), 322–334. https://doi.org/10.1080/07317131.2011.574519

Last month, we finalized the metadata for some additions to a large digital collection of material about Cuban and Cuban-American theater.  This particular collection has multiple different fields for creators and contributors (e.g. author, director, cast, costume designer, etc.).  Additionally, this collection has been on-going for several years, meaning many people have created metadata for it.  Now seemed like a good time to exert some authority control on the names in those various creator and contributor fields, doing some standardization and normalization of names.  We use the authorized form of names from LCNAF, for names that have authority records, so my goal was to make sure that all names with LC records used the authorized form, and for all other names, to use one consistent form across all of the fields.

Aaron Burr really cares about name authorities.

CONTENTdm, which we use to host our digital collections, has an option for individual metadata fields to be set as a “controlled vocabulary,” meaning that we can define a list of terms that are considered valid for that field.  Setting the fields I wanted to work on for this project as controlled, with the existing contents of the field used as the list of approved terms, meant that I was able to export a text file for each field with all of the terms it contains. CONTENTdm saves those files on its server, so I was able to easily use FileZilla to download the text files for the 13 fields in question.  

I wanted to combine those 13 lists of names into a single list, but add a second column with the original filename, so I could track which names came from which fields. A little googling, and a suspicion that this should be easy to do on the command line, brought me here, and I adapted the script slightly for my purposes: for /f %a in (‘dir /b .txt’) do for /f “tokens=” %b in (%a) do echo %b;%a >> all.txt

Tah-dah! I now had a semicolon delimited file with columns for the Name and Field. The next step was to de-dupe that list - I needed to find names that were in multiple fields, combine the values from the Field column into one cell, and remove the duplicates. I tried to do this in Exel, and it was a struggle. I briefly considered writing a macro to do it before giving up and doing it manually.  I’m glad I didn’t waste a lot of time on a macro for it, because I later realized that it was very simple to do in OpenRefine (thanks to reading Ruth Tillman’s excellent article in the most recent Code4Lib).  The way to do it in OpenRefine, for future reference, is:

• Sort by Name, and select “Reorder Permanently” in the Sort menu
• In the Name column menu, under Edit Cells, select “Blank Down”.  
• In the Field column menu, under Edit Cells, select “Join multi-valued cells”.  That will combine all of the values into one cell, using a selected character as a delimited.

So much easier than an Excel macro

Okay, now the fun part: I had a list of 740 names and what fields they were found in, so I wanted to:

1. Reconcile the list of names against LCNAF to make sure we are using the authorized form, if there is one, and
2. Identify variant forms of the same name, particularly those that aren’t established in LC.

Fortunately, OpenRefine can help with both of those things!

I tackled reconciliation against LCNAF first.  I’d previously tested out a VIAF reconciliation service developed by Jeff Chiu, and had good luck with it.  It’s fast, and seems to return good results.  I copied the list of names to a new column in Refine, and ran the reconciliation service on that column.  After running the reconciliation, I went through quite a few names that had potential matches and manually confirmed (or not) the match, particularly for names that didn’t have qualifiers like dates.  I ended up with 98 names that were matched to LC authority records.  I created a new column, with only the names that were reconciled with VIAF, using the GREL expression cell.recon.match.name.

My next step was to use the clustering function in Refine to identify potential variants of the same name. I knew there were a lot of potential variants: names missing dates, missing diacritics (or extraneous diacritics), typos, etc. I briefly tried to do this work myself, and quickly realized it was extremely difficult to do. But the power of algorithms came to my rescue! I created another new column of names, and used OpenRefine’s “Cluster and edit” function on it (found under Edit Cells). The clustering function identifies potential variants that represent the same thing and lets you quickly edit them all to one version (or to something else altogether). I tried out various methods of clustering, and found that they caught different potential variants. For each cluster, I chose the correct form and edited the cells to include it. In many cases, this involved going back to the items themselves to verify if, for example, this person used an accent mark. This was a time-consuming process (possibly the most time-consuming single step), because it involved so much looking back at the materials to determine the correct form.  However, it wasn’t difficult, and it was SO much easier than if I’d tried to do the matching by hand.

Diacritics gone wild

With all of the reconciliation and clustering done, I had my list of names that needed to be edited. I exported the project out of Refine and back into Excel, with colums for the original name, the reconciled name, and the clustered name. I used some IF(ISERROR(MATCH())) formulas to see if the original name matched the values in the reconciled and clustered name columns, and from there was able to separate out the names to be changed.  I ended up with a total of 88 names.  I then added columns for each original field to indicate if the name was found in that field, allowing me to filter for the names to change in each field.

Look at all those names to be updated!

Last but not least, I went into CONTENTdm and, field by field, edited the controlled vocabulary and used the built-in “Find and replace” tool to update the names that needed to be changed.  Because I had a well-organized list of what names needed to be changed in each field, I was able to move through this step fairly quickly.