Deleting Items from SAGE Solr

SAGE stores documents based on the id set in the Reader and Writer of the application. This is often based on an id of the original source application. Occassionaly, that id pattern will change. When you do an update, it will create brand new records. SAGE has no mechanism on how to delete the old duplicate records. This document explains how.

A Solr Delete Script

On your local machine, write a file like the one below. It’s usually better to write locally as we cannot guarantee our pod will have a text editor (or one where I have all the commands memorized).

#!/bin/bash

# Configuration
SOLR_URL="http://localhost:8983/solr/sage-core"
ID_FILE="ids.txt"

# Optional: check the file exists
if [[ ! -f "$ID_FILE" ]]; then
  echo "Error: file '$ID_FILE' not found"
  exit 1
fi

count=0
errors=0

while IFS= read -r id || [[ -n "$id" ]]; do
  # Skip blank lines
  [[ -z "$id" ]] && continue

  response=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST \
    -H "Content-Type: application/json" \
    "${SOLR_URL}/update" \
    --data-raw "{\"delete\": {\"id\": \"${id}\"}}")

  if [[ "$response" == "200" ]]; then
    echo "Deleted: $id"
    ((count++))
  else
    echo "ERROR ($response): $id"
    ((errors++))
  fi

done < "$ID_FILE"

# Commit the deletes
echo "Committing..."
curl -s -X POST \
  -H "Content-Type: application/json" \
  "${SOLR_URL}/update" \
  --data-raw '{"commit": {}}'

echo ""
echo "Done. Deleted: $count | Errors: $errors"

Save the file something like delete_docs.sh. Then:

chmod +x delete_docs.sh

Notice, that the file references a text file called ids.txt.

Also create a file with all the ids you want to delete on separate lines.

Copying to the Server

Copying with kubectl is easiest. Assuming you have it installed and set up (see instructions elsewhere here), switch contexts:

kubectl config use-context prod-apps-cluster

Then copy files to the pod where you will run. You’ll need to identify pod name with kubectl or via Rancher directly.

kubectl cp ids.txt sage-solr-7fc7c9986b-tmf7k:/tmp -n sage
kubectl cp delete_docs.sh sage-solr-7fc7c9986b-tmf7k:/tmp -n sage

Run Script

You can run the script locally with kubectl, but assuming you’ve copied like above, you can just connect to the pod using your method of choice and run it:

./delete_docs.sh

If successful, this will look like this:

solr@sage-solr-7fc7c9986b-tmf7k:/tmp$ ./delete_docs.sh
Deleted: Mi0yNWRkNDVmYS04NWZjLTRiYjMtODE0ZS05NDQzYzg2ZWZlM2Y=
Deleted: Mi1kNzk4NjhjOS0yZDA2LTQ0M2UtYTBlYy1lM2E5NmFmZjk3ZTM=
Deleted: Mi1lZmZkYjMxMS1mM2VlLTQ3MTItYjNmZS01MjgyZDkxZmNkYTQ=
Deleted: Mi0xODc1N2FlZi0wN2JkLTQ1ZWYtYWYzNS1iMjFmZjJiNDM2OTM=
Deleted: Mi03OTI0ZWY0OS01ODM0LTRhNWItYTE5OS0wMWVmYWNhYjcyOTg=
Deleted: Mi02N2RmZTkzNi1hNTI0LTRlMWYtYmU1Mi0yZWQzMzI2NGVjZjM=
Deleted: Mi1jY2Q4OGU4Mi1jNGE3LTQ3NTgtOWMzZC05OGY4NTg2MmViNDY=
Deleted: Mi0zNmJmODBkNC1iMTFhLTRlNWQtYmNhMi03NGE5ZmY5YmEyYzU=
Deleted: Mi00YzI1MTEzMy02YmI3LTQ5YmQtOTcwMi05Njk4ZjlkMTNmOWE=
Deleted: Mi01NTk4NjNhYi0wNzkzLTQxMzctYjJkOC1jZDhkZjllZjhmMjk=
Deleted: Mi05ZmRjZTMzZS02OWIzLTRjNjMtYmViZC1lNTJlNDNiYmNkZmU=
Deleted: Mi03YjBlODcxYi1kOTEzLTQ4MmYtYjkxYy0wYmQ1N2EzNzcyZDg=
Deleted: Mi0zZjgyNWU4NS03N2ViLTQxMDUtYTcxOC1iNTk0NGI5MmEwZTg=
Deleted: Mi0xNTk3OWFiMS1hMTMzLTRkZTYtOWNlMS02NGRiMjNmNWFlMzE=
Deleted: Mi1iYjZjZWE4Zi05YjZlLTQwZTAtOTEzMi1lNzg4MTg4MGRmYzc=
Deleted: Mi05ZjkyNzQ1ZC02YTNhLTRmOWQtYjQ2ZC0yYzVjMTE5NTI2ZGU=
Deleted: Mi1hOTlkNTA0ZS0zODllLTQ2ZTgtODhmNC1hMjU0YTlhMjdlYmM=
Deleted: Mi0xNDAwMTJlOC0zMjhkLTRjYTUtYjEwZS1lYjQzNWY1ODJkYjE=
Deleted: Mi1mZWRmOWI4OC1hNTU0LTQxZGItOWMwOC1hYWYwZGNjZWY3ZTY=
Deleted: Mi03OWEyZWZhOS0zMTJiLTQ1ZGUtYmM2NS00ZmI0ZWU4OTA1YWY=
Deleted: Mi1mYTYzM2FiMy1kYzM4LTQ3ZmMtOWE4ZC05Y2FjODQxNjc1MjI=
Deleted: Mi03MWMzMDBhMi0wZDIyLTQ3YWEtYWJmYS1mZWRkNDcwZjFhNWU=
Deleted: Mi1mMjI1MTVlNS03ZmNjLTRiNzktYWY4ZC04OWZjYmYzNmVmNTk=
Deleted: Mi04Y2MwNGNkNC1lNDNhLTRlN2EtOTEyZi1hNTA2YTkxNTlkNzE=
Deleted: Mi03Mzc2M2RkNy03YzA0LTRjYzAtOGNkMS1kNDk5NjdmMWFiYjc=
Deleted: Mi03NzMwMmFlZS03ZWZjLTQ2OGEtYTViYy04ZTFkNDAxOTJhMzI=
Deleted: Mi0xOTM3ZmE2Yi1kMzYwLTRlZjctOTA2My1mZTZiNjc3ZWFkMDk=
Deleted: Mi01MTgzMGI0MS00NzMzLTQ0OTItYjE2YS1iMDMzYmI0ODFhZjQ=
Deleted: Mi1hNDU0NjMxZS1kYjY1LTRiNjgtYWEyMC0yZWEwMzZjZjViMDA=
Deleted: Mi1hNGI3MjA0MC0zZDUwLTRkZWQtODYwMi0zYzUxODA1NDhiMDc=
Deleted: Mi1lZjg0NzVlZi0yMDU1LTQ0OTYtOGRkYy03YzdhMDIzYjM5ZDI=
Deleted: Mi00ODkwMmMwNC0zOTkzLTQwODQtOGNjZS1lMzFkZjUwNjkyMGE=
Deleted: Mi1kMWM0ZGE1ZS01YjZjLTRkZDYtYWEzOS04OTY0NTU1NGIzN2E=
Deleted: Mi1lNjdkMWRhMS1kOWU5LTRlNTQtODQyZi0zOTIyNmE2ZWU5N2Y=
Deleted: Mi1lMDJhMDEyYi1lNzBjLTRhYzQtOTEyNi02Y2UwMWVkNTIwYTA=
Deleted: Mi1hOTJlNjYxMy1iMDVkLTRkZjktODNkNi0zMGNjOThkZTFhYzg=
Deleted: Mi01ODJhYjliZC0xMTM0LTQ1ZGYtYTA0NC00MjAxOWZhMDhlZTQ=
Deleted: Mi1kMGQwMTQzZC0wNjUxLTQ0NDEtYjdmZC0zNDQ0NTQ0YjJmMWE=
Deleted: Mi0yM2FiNmEwMy01M2NkLTRlMGItODA1Yy03ZTEyOTkwMTc2M2E=
Deleted: Mi01YjQ2ZmVhZS0xM2U3LTRhOWMtYWQ0NC0wNjA4ZTlkNGJiMWQ=
Deleted: Mi1jODk1Mjg4Mi05MGM4LTQ0OTYtYjRhOC0wMWI1MjJmNzZlNWY=
Deleted: Mi1jMDEwZjVkMS00OGVhLTRiOGQtOTg1Zi1kNzZjMzAxZmQzOWQ=
Deleted: Mi0zYzdkNmQzZS04NWYzLTRhZjAtYjYyOC1mZWM3ZWU0ZWY3NWQ=
Deleted: Mi0xNTJmMzEwZC1lMjIxLTQzMDktOGU2Yy0wNDJiODMxN2IwYzY=
Deleted: Mi1hYzdlMWQ5NS1jOTM1LTQ2OGEtOGJlYy0xNmUwOGNkOGExN2Q=
Deleted: Mi00Nzk2M2FiMi0xNWVlLTQzYTktYTk2Ni1hNTQwNDUxNDI0NDM=
Deleted: Mi0zMzRlNjc2NS1kOWE0LTRhZjEtOTdhYS0zMTIzMGZlNWE1MWQ=
Deleted: Mi01OTk4MWU4OC0wMjY2LTQ1MWQtOGVhZi0xMmZlNzA2ZTY2OWM=
Deleted: Mi1iYWQ3OGVmNy1jZGY1LTQxMWEtOTgzMS1mNjdkN2Y3Njg2MmM=
Deleted: Mi01M2FmMjQzYy02YjY3LTRhZWMtYWI0YS01NTE2MjNjOTQ0MWE=
Deleted: Mi1jN2MxNGViNC05MTg3LTRlYmYtYjc2OS1hOGIzZDk3YjVhNmU=
Deleted: Mi01YTk3MTM2MC1mMWI5LTQ2ODQtYjJkOS04ZjEzN2Y1NDlkNzY=
Deleted: Mi1iZjRkNmFhNC00NjIwLTRlODQtODBmYy1lM2MzZGEwOWNmNzc=
Deleted: Mi1kNzBlMWQyZS04OWU0LTRhODMtYjMzOC03MjlmZjQ0NTRjYTg=
Deleted: Mi04ZmQ2YWVhMC0yNGMyLTQ5MDAtODRiOS1kOTM3YjZiNWU3ODM=
Deleted: Mi1iOGQ3NGVkNy1kODY5LTRlMzgtOGU1MS1lMWMxNjcyZGRlNTE=
Deleted: Mi1hYWUzNjdhNC1iNjk0LTQxNjktYjcwMC00NzRjMDliNDc3ZDM=
Deleted: Mi1kNWY3YTA4OC02NjVjLTRhZDgtODg0MS00OTg3OGMzNjljM2I=
Deleted: Mi00ZGExMTc5YS0zNzc0LTRlMzUtODIxOS1kNzViZDI2ODVmNzM=
Deleted: Mi1hZmQ2OWI1Yy1lODdlLTQ5OTYtYmNmMC1iNjgyNGY2NDhhZGQ=
Deleted: Mi0xZGM1NmI3Zi0xOGU2LTQ5NjQtODhmMy1mYzZiOWY1ZDA2YTQ=
Deleted: Mi02OTE0ZWQ4OS1hNzk2LTQ5ZDEtYjA4ZS1jYzRmY2JjYTQ5ZjA=
Deleted: Mi1mMmQ3Njk3YS1hMjBhLTQ5ZGMtYjAyZS02MWQ5NTEwNTcwNmU=
Deleted: Mi01Njc4ZDUwNi0yZjc2LTQwZWEtYmI5Yi0zOWQ0ZGZiMTU2ZWM=
Deleted: Mi01ZjE3ZGY0Yy0zYTU3LTQ2MTItOTI4Ny0wYmI1OGJmMGVmMDk=
Deleted: Mi02M2VlNzU5Yi1mNGZlLTRmZjItYWZlZC02ZTdhY2M5ZDAzYzI=
Deleted: Mi05ZGI4NmJhYy1kNGQ0LTRiOGUtODkxMy04YjUyMjIwOGZmNmY=
Deleted: Mi0wM2NiZmMwYy02OGE2LTRkODItODQyNi02N2VkNGEyODk1MWQ=
Deleted: Mi00MGUwOWM3Zi1hZGY1LTQyMTItYmM3ZC1kZjllNTA0NWM4NTY=
Deleted: Mi04OWQ4ZjcwZS00ZGMzLTRjZGEtYWE3MC02MjQwMTVkM2YwYmY=
Deleted: Mi1mNGE4ZmZhZi00ZDY4LTQxNzUtOWE0NS1hM2VmZGQ5OTQwYzY=
Deleted: Mi01NmUzZjk0MS0xYzg3LTRkZWYtOGYzZi0yYmEzZWU4ZGE0ZjQ=
Deleted: Mi1iZDJlZTFmMC1lOThmLTQ1MjgtODU4OC03YzY4M2QzNzBjOGE=
Deleted: Mi02ZDY4MmJjMy1jNzQwLTQwMTItYWJjZi00ZGMyNWNhNGNlZTU=
Deleted: Mi01ZGI4NjkxMS1lZWMxLTQzYTEtYjUxZi0zZTZmODM3NWJlZDI=
Deleted: Mi0yZmIyYTZkNS02M2Q4LTRiZTMtOWI4YS05NjI4ODNmZDBhMmI=
Deleted: Mi00ODQyMjcwMy01M2FiLTQ2NTMtOTBlZi1kMDEzNTM2MjIxNTY=
Deleted: Mi1kOGY4NzhkOC0wZWJmLTQ1YWMtYmYwOC0zMDMzYTk4MTVjZTc=
Deleted: Mi04MzAzZTg0Ni05ODVlLTQ4NmQtODhlOS05NDljYTY3OWZlZDY=
Deleted: Mi00NDU2M2JjOC1lMjQ0LTQ2ODYtYmNiMS1iMDZjZmE3N2NiMWQ=
Deleted: Mi1jYTg5MzAxYy03ZDNlLTQ5MDYtYjlhNS00MjMzMWUzNTFlZDE=
Deleted: Mi02ZGViOTk3Ni1mMDFmLTQ5OWYtOWJmNC1kYTE1NTBmNDc4NTA=
Deleted: Mi0yYjFlMTU4MC05Y2I2LTQ4MDctYTUyMi1lZTNhOTZlNzdkYjU=
Deleted: Mi0zMWU3NWIzMS1lZGJiLTRlYzgtYTdhOS00ZTQwYzlmOTM2Y2Q=
Deleted: Mi1jNjA5MTkxMi03NDg4LTRjNzYtODk3YS0zYWQyMTQ5NzE1NjI=
Deleted: Mi03YmEzNGY2OS05N2ZkLTRmNjUtYjYxMS1mM2UwNTFmZDAxMDY=
Deleted: Mi1lYTZiOGFkZS1jZmMxLTRiNDEtYmY5ZS02MzIzMDQ1YTk3Zjc=
Deleted: Mi01M2JjNmJmYi01NGQxLTQxNDQtOTAyZS05ZTM1ZjcxNWVlMDg=
Deleted: Mi0xZjY4OWNkZC04NTQ4LTRmNDMtOWJmMS03YjFlYjMzZmM0MzA=
Deleted: Mi0xMzQ4MzA0ZS1mNTAyLTQxODMtOWJiYi1kNTIwZDIxNTZmZjM=
Deleted: Mi0yNGM3MWQwZC05MjE1LTQ0ZTAtYWJiZS0zZGFjZDRmMjU1MTI=
Deleted: Mi04ODdmNWEzNy1kMTdjLTQyY2YtODY5Mi04NzllM2I4NGNkY2Y=
Deleted: Mi0xZjg4YzFkMy1kNmYzLTQ1YTMtODgwNy05OGQ5NWVjN2Q0NWM=
Deleted: Mi02ODExZTRkZi1mNTg4LTQ3NjctYWMzYy1jNWM3MmY5NDUxNzE=
Deleted: Mi00NjUxNzkxZC0zODI0LTQ4ZGQtYmZiMy1jYmM5ZjViODNjMWY=
Deleted: Mi1jYzljZGFjYy04NjdiLTRjMzYtODlhMy01Mjk3ZmNlYTc2ZGY=
Deleted: Mi02ZmVmMWU3MS0yMjg2LTQ3OTEtODNmNC01ZTk2YjY2NTRmZTI=
Deleted: Mi03ZmE2MzkyMC0zOWE4LTQwMDgtYjQ1OS02MTVjYTE2ZjUyODY=
Deleted: Mi1jODNjNzAzMC04NjBlLTRlMzUtYmQwNS0yMDUyMzc5ZWEyMmM=
Deleted: Mi1mZmE5NWNjNy03ZjAzLTQzNDQtODQzMC1hNDBjNzg3MGRiZjI=
Deleted: Mi03OTRmNWEzZS1hYjllLTRlNmYtOTRkNi1jNzE4MmM1ODg2OTc=
Deleted: Mi04ZDBjZjQyYS05MjFkLTQxNDMtOWRjOC00NTJiNjZhY2ZkMTE=
Deleted: Mi04YTVlYzE1OC1mZTAwLTQzMGYtOTc4YS02MzE1MTNmMmIyYzI=
Deleted: Mi1jYzMxZTI4YS02ZjE3LTRiMDMtOTJmYS1hNWI4YmNiMDgwNGM=
Deleted: Mi0yYThhZmY5My1kMDVjLTQwZDItYTZiNS0yNDczZjk1MzdhNTA=
Deleted: Mi0zYThmYTBlYS03NTY3LTQ3YTEtYTBjMy01Y2M2Y2E2ZDE3MGM=
Deleted: Mi1hMDk5OGQ3Ni0xOGIyLTRlMjItYjZmYi02ZGIzZDU3MGQwZjY=
Deleted: Mi1mNDg1YjMzMi1hMzAxLTQ5OGQtYmUyNS04YWYwYmMyZTExZDQ=
Deleted: Mi1mMzNiNGI2Mi1hNmExLTQyZjQtOTEwMy0xMmUyODRkMjAyNDY=
Deleted: Mi01MWQ1YWYyZi1hYjdhLTQwMzktYTM3YS05MjNiY2Y2MzcxZTc=
Deleted: Mi1lNzBhNzdjMi1kMDRkLTRmMGMtOWUzMy0yOGYzNWUxOTUwMWM=
Deleted: Mi03MzNlMDE1OC0zNDc5LTQ5ZmEtYTk0NS1lY2VjMGQ0ZDhhNDU=
Deleted: Mi04ZTdlNjJlNi1hYWI4LTQxNzUtODkxZC05OTZjZDlmMjE3NzU=
Deleted: Mi05OGE1OWE4MS00MTY1LTQ5YjktYjgwZi02Mjg4YzAyYzQ2NjQ=
Deleted: Mi1lMmNmYzBkYi1mYjI2LTQ0NDUtYjVmMy0wYWJiM2E3MmFjZTY=
Deleted: Mi1jYzhiNzk2ZC1lMTE0LTRkNDctOGU0OS1hMGU1ZTVkZjQ5NTI=
Deleted: Mi1mYzM0NTEyMS05Y2ZhLTQ0MGYtOTYwOC02ODc4MjNiZDQ3ZTE=
Deleted: Mi1iOWI1M2Q0OC00ODc5LTRiZDYtYWY0NS01ZTk2ZmU3ZWMzZmQ=
Deleted: Mi1iZDg5YWNlNS03MjMwLTQzMDktYjM3MC1mZGQxZTlmMzA2ZWE=
Deleted: Mi1hNTRjNTU2YS00YjFkLTRjZTctYjA4Yy0zYzE2MzMzZDZhNTg=
Deleted: Mi1iYTFhOTMxZS0yY2VkLTQ5OGYtYmUxNy1jYWRjYTY2NGY2YjM=
Deleted: Mi1lZjUzNGQ1OC01YzY2LTQ5MmUtYTE2MC02Yzc5N2FjMGIwMjU=
Deleted: Mi1mNTVlMDY2NC1jOTA3LTRkOTYtYjMxZC0xY2FkNjBiYWIwMmU=
Deleted: Mi0yZDVhN2E3NC0xMjY2LTQ4YzEtYmE0Zi1mY2JhNDY3MGJjNjA=
Deleted: Mi1kODI0ZTc5My0zM2I5LTQ0MzYtYmE4Mi0xMzU0Nzk2MTc5Zjg=
Deleted: Mi0zMjVhN2EzYy0yYTVjLTRlMWItOThhMy1kZDNlMjUwMGE1YzI=
Deleted: Mi01NDdmOTcwMS0wYjU2LTRlZGEtODdhYy0yZDhmYzEwZjY3ZjA=
Deleted: Mi0xZjllMWU5Zi1mNmUyLTQ3MTAtOTNiNy0yZWJmYWZhMTVmNmY=
Deleted: Mi1iZmEwOGYzOC02MTE2LTQ2NDMtYjNjMS0yNGUwZmY1NDgzNGI=
Deleted: Mi0zNTE3ODNjMC0yYmQxLTQ1YzItYTc3ZS01YjM1ZmZiNWM1YjE=
Deleted: Mi02NzM2YjY0MS00OTM3LTRmZjYtYjgwMy02YWM1YTJiMDFkNWM=
Deleted: Mi1lNWQ3MjNkMC1jNzYyLTQ2OWEtYWYzNi0xOWY0Y2NhMjEyZDY=
Deleted: Mi0zNTg1YTRhMi1iMjlmLTQ2MWUtOTYzZC0yNGE0MTdhNTc0MDc=
Deleted: Mi0zNzdiMTc1NS1mNTk1LTQ2MmYtYjc3YS00ZDkzMTBhYTU1NDA=
Deleted: Mi1lMGU0Zjk2ZS03ODQzLTRjZWMtYTVmZi01NDQzMzg2ZjBhNzA=
Deleted: Mi1jODViYmRlZS02OTk3LTQ5ZTEtYmVkYy00ZTdhNjg3MTdiNmQ=
Deleted: Mi01YTM3NWU1ZS0wNzYxLTQ1MTAtYTJlMi05YmU0NmZlZTFlZDg=
Deleted: Mi02M2I3ZjExYi1jNWIzLTRmZTItOTk2NS1mYmY2ZGQ0OWQ4MWQ=
Deleted: Mi1lZTQ5ODk5Yy0zMTc4LTRlYzQtYTQ2Yy1mMTgwY2FkZjQ4ZWM=
Deleted: Mi1hOTgxYjYxYS0zMmQ3LTQ5MDEtYjdkMS0xZTBhMjhjZmNlNWE=
Deleted: Mi0zZjk0MzdlYS0wNjdiLTQyMDUtODk5YS0zMTE3NjY4NzQ3OTQ=
Deleted: Mi02NjBjOGRiOC0wYjE2LTQ0ODEtYmU2OC04MmU0Njk1MDllNzU=
Committing...
{"responseHeader":{"status":0,"QTime":43}}

Done. Deleted: 142 | Errors: 0