Skip to content

Commit d043ee4

Browse files
committed
Restore student data
1 parent 387531a commit d043ee4

File tree

10 files changed

+261
-49
lines changed

10 files changed

+261
-49
lines changed

README

-45
This file was deleted.

README.md

+94
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# The UNL faculty staff and student directory.
2+
3+
The UNL_Peoplefinder package contains all the source code for the UNL directory.
4+
Additionally, this package provides an API which developers can use to retrieve
5+
directory information and perform searches.
6+
7+
## Maintenance Tasks
8+
9+
The Officefinder "yellow-pages" portion of the site is stored in a mysql database.
10+
11+
## Editors:
12+
13+
Editors can be added for any yellow-pages listing and permissions are
14+
hierarchical. We expect
15+
16+
When the print "Centrex" directory stopped being published, we used the HR
17+
Contact List (go.unl.edu/hrcontacts) to assign a default user to each
18+
department. This list should be updated on a quarterly basis.
19+
20+
The script to update permissions is `data/setup_permissions.php`
21+
22+
Once we have accurate editor information, it's best to work with Linda Geisler
23+
and send a note to all the HR Contacts (SAP coordinators) via the
24+
`scripts/mail_sap_coordinators.php` script.
25+
26+
## Data sources
27+
28+
This project uses and combines data from many different sources
29+
30+
## HR and orgunit information
31+
32+
Jim Liebgott maintains a process which exports the hr_tree.xml file. This file
33+
contains the UNL departmental hierarchy, which is used in the directory.
34+
Jim's process connects to CSN's data warehouse and exports the XML data.
35+
36+
The script `data/update_hr_tree.php` runs nightly (via cron) to find new or
37+
moved units and add/update them in the database.
38+
39+
The script `scripts/flag_sap_orgs_without_employees.php` runs nightly to mark
40+
units which have no employees or child units and hide them from the directory.
41+
42+
## Faculty data
43+
44+
Faculty data is retrieved from Activity Insight. There is a cron job that runs regularly to refresh that data. This data is stored in the cache.
45+
46+
```
47+
php scripts/cache_knowledge.php
48+
```
49+
50+
## Active Directory
51+
52+
Most of the data for faculty, staff, and students is retrieved from Active Directory via LDAP queries.
53+
54+
## IAM Views
55+
56+
The active directory data does not contain all of the information we need (email and employment appointments for example). We augment the active directory information via Oracle database views provided from Identity and Access Management.
57+
58+
## Student data
59+
60+
Student data is provided by Terry Pramberg (Student Information Systems). He provides us with .txt files (which are actually csv files) that contain public student directory information. There are two files:
61+
62+
1) `data/unl_sis_bio.txt` which contains the NUID and class level for students.
63+
2) `data/unl_sis_prog.txt` which contains the majors for each student.
64+
65+
These files need to be manually uploaded to the production server on a regular basis.
66+
67+
There is a script that combines these files and caches the result in memcache for each student. As data is retrieved from the directory, these results are merged with their appropriate Active Directory records.
68+
69+
To update this data, run
70+
71+
```
72+
php scripts/import_sis_data.php
73+
```
74+
75+
Note: this needs to be ran at least once a day to keep the data in the cache
76+
77+
## INSTALL
78+
79+
1) Run: 'npm install; grunt; composer install'
80+
2) Copy www/config-sample.inc.php to www/config.inc.php and add your LDAP credentials or uncomment the webservice driver line
81+
82+
83+
## Rebuild cache
84+
85+
```
86+
php scripts/rebuild_cache.php
87+
```
88+
89+
This will:
90+
1) flush the entire cache
91+
2) populate the cache with SIS data
92+
2) populate the cache with faculty data
93+
94+
Other data will be cached as when data is retrieved from Active Directory and the IAM views.

data/unl_sis_bio_sample.txt

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
"EMPLID","CLASS_LEVEL"
2+
"00000000","GR"
3+
"00000001","SR"
4+
"00000002","SO"
5+
"00000003","JR"
6+
"00000004","FR"
7+

data/unl_sis_prog_sample.txt

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
"EMPLID","INSTITUTION","ACAD_PROG","PROG_DESCR","PLAN_DESCR","ACAD_PLAN_TYPE"
2+
"00000000","NEUNL","GRD-G","Graduate","Entomology","MAJ"
3+
"00000001","NEUNL","ARH-U","Architecture Undergraduate","Landscape Architecture","MAJ"
4+
"00000001","NEUNL","ASC-U","Arts & Sciences Undergraduate","Biological Sciences","MAJ"

scripts/import_sis_data.php

+119
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
<?php
2+
require_once __DIR__.'/../www/config.inc.php';
3+
4+
$cache = UNL_Peoplefinder_Cache::factory();
5+
6+
// This is the array that will store all of the data
7+
$students = array();
8+
9+
// First, load the bio info and create the initial structure
10+
$csv_bio_file = __DIR__.'/data/unl_sis_bio.txt';
11+
12+
if (!file_exists($csv_bio_file)) {
13+
// You must manually upload the file
14+
return;
15+
}
16+
17+
$csv_bio_file = file_get_contents($csv_bio_file);
18+
$csv_bio_file = mb_convert_encoding($csv_bio_file, 'UTF-8'); //See note 'Parsing export from drupal webforms
19+
$csv_bio_rows = explode("\n", $csv_bio_file);
20+
21+
// Remove the first row, which should be the column headers and not actual data.
22+
array_shift($csv_bio_rows);
23+
24+
foreach ($csv_bio_rows as $row) {
25+
/**
26+
* [0] = NUID
27+
* [1] = CLASS_LEVEL
28+
*
29+
* ONLY non-ferpa protected records are included
30+
*
31+
*/
32+
$data = str_getcsv($row, ",", '"');
33+
34+
if (!isset($data[1])) {
35+
// This is likely a blank line at the end of the file
36+
continue;
37+
}
38+
39+
$students[$data[0]] = array(
40+
'unlsisclasslevel' => $data[1],
41+
'unlsiscollege' => array(),
42+
'unlsismajor' => array(),
43+
'unlsisminor' => array(), // Minor data is not shown (not listed as public directory information in the policy)
44+
);
45+
}
46+
47+
// Second, load the college, major, and minor information and add it to the existing structure
48+
$csv_bio_file = __DIR__.'/data/unl_sis_prog.txt';
49+
50+
if (!file_exists($csv_bio_file)) {
51+
// You must manually upload the file
52+
return;
53+
}
54+
55+
$csv_prog_file = file_get_contents($csv_prog_file);
56+
$csv_prog_file = mb_convert_encoding($csv_prog_file, 'UTF-8'); //See note 'Parsing export from drupal webforms
57+
$csv_prog_rows = explode("\n", $csv_prog_file);
58+
59+
// Remove the first row, which should be the column headers and not actual data.
60+
array_shift($csv_prog_rows);
61+
62+
foreach ($csv_prog_rows as $row) {
63+
/**
64+
* [0] = NUID
65+
* [1] = INSTITUTION
66+
* [2] = ACAD_PROG (program code "ARH-U" for example)
67+
* [3] = PROG_DESCR (college)
68+
* [4] = PLAN_DESCR (major/minor name)
69+
* [5] = ACAD_PLAN_TYPE ('MAJ' => 'Major', 'MIN' => 'Minor', 'COS' => 'Course of study')
70+
*/
71+
$data = str_getcsv($row, ",", '"');
72+
73+
if (!isset($data[1])) {
74+
// This is likely a blank line at the end of the file
75+
continue;
76+
}
77+
78+
if (!isset($students[$data[0]])) {
79+
// Student is not tracked, likely do to a FERPA flag
80+
continue;
81+
}
82+
83+
$students[$data[0]]['unlsiscollege'][] = $data[2];
84+
85+
if (isset($data[5])) {
86+
$plan_type = strtolower(trim($data[5]));
87+
88+
// Display Course of Study as a major per Steve Booton
89+
if ($plan_type === 'maj' || $plan_type === 'cos') {
90+
$students[$data[0]]['unlsismajor'][] = $data[4];
91+
} else {
92+
echo 'unknown plan type ' . $plan_type . PHP_EOL;
93+
}
94+
}
95+
}
96+
97+
// Clear all old memcache results if needed
98+
$existing_keys = $cache->get('unl_sis_keys');
99+
100+
// Save to Memcache
101+
$prefix = 'unl_sis_';
102+
$unl_sis_keys = array(); // Keep track of all existing keys because Memcached->getAllKeys() is not reliable
103+
foreach ($students as $nuid => $data) {
104+
$cache_key = $prefix.$nuid;
105+
$unl_sis_keys[] = $cache_key;
106+
$cache->set($cache_key, serialize($data), false, true);
107+
}
108+
$cache->set('unl_sis_keys', serialize($unl_sis_keys), false, true);
109+
110+
111+
if ($existing_keys) {
112+
$existing_keys = unserialize($existing_keys);
113+
foreach ($existing_keys as $key) {
114+
if (!in_array($key, $unl_sis_keys)) {
115+
// only remove keys that don't exist in the new data (avoid a brief moment when data might be unavailable)
116+
$cache->remove($key);
117+
}
118+
}
119+
}

scripts/rebuild_cache.php

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
<?php
2+
require_once __DIR__.'/../www/config.inc.php';
3+
4+
$cache = UNL_Peoplefinder_Cache::factory();
5+
6+
// Clear the existing cache
7+
$cache->flush();
8+
9+
// Import SIS data
10+
require_once __DIR__ . '/import_sis_data.php';
11+
12+
// Cache faculty data
13+
$driver = new UNL_Knowledge_Driver_REST;
14+
$driver->getAllRecords();

src/UNL/Peoplefinder/Driver/LDAP.php

+2
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ class UNL_Peoplefinder_Driver_LDAP implements UNL_Peoplefinder_DriverInterface
5555
'department',
5656
'unlHROrgUnitNumber',
5757
'departmentNumber',
58+
'unluncwid', // the NUID
5859
);
5960

6061
/**
@@ -95,6 +96,7 @@ class UNL_Peoplefinder_Driver_LDAP implements UNL_Peoplefinder_DriverInterface
9596
'unlSISPermState', //missing
9697
'unlSISPermZip', //missing
9798
'unlSISMajor', //missing
99+
'unluncwid', // the NUID
98100
);
99101

100102
/** Connection details */

src/UNL/Peoplefinder/Driver/LDAP/Entry.php

+20
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ protected static function normalizeEntry(array $entry)
2424
{
2525
$entry = self::fix2018LdapToAdChanges($entry);
2626
$entry = self::fix2017LdapChanges($entry);
27+
$entry = self::attachSISData($entry);
2728
$entry = UNL_Peoplefinder_Driver_LDAP_Util::filterArrayByKeys($entry, 'is_string');
2829
unset($entry['count']);
2930
foreach ($entry as $attribute => $value) {
@@ -170,6 +171,25 @@ protected static function fix2018LdapToAdChanges(array $entry)
170171

171172
return $entry;
172173
}
174+
175+
protected static function attachSISData($entry)
176+
{
177+
$cache = UNL_Peoplefinder_Cache::factory();
178+
179+
if (!isset($entry['unluncwid'][0])) {
180+
// No NUID provided
181+
return;
182+
}
183+
184+
$sis_data = $cache->get('unl_sis_'.$entry['unluncwid'][0]);
185+
186+
if ($sis_data) {
187+
$sis_data = unserialize($sis_data);
188+
$entry = array_merge($entry, $sis_data);
189+
}
190+
191+
return $entry;
192+
}
173193

174194
public function append($value)
175195
{

src/UNL/Peoplefinder/Record.php

+1
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ class UNL_Peoplefinder_Record implements UNL_Peoplefinder_Routable, Serializable
5454
public $unlSISMajor;
5555
public $unlSISMinor;
5656
public $unlEmailAlias;
57+
public $unluncwid; // NUID
5758

5859
protected $knowledge = false;
5960

www/templates/html/Peoplefinder/Record.tpl.php

-4
Original file line numberDiff line numberDiff line change
@@ -83,10 +83,6 @@
8383
<?php endif; ?>
8484
<?php endif ?>
8585

86-
<?php if ($context->isPrimarilyStudent()): ?>
87-
<p>Due to transitions in data systems, information pertaining to student rank, major., etc. is temporarily unavailable.</p>
88-
<?php endif; ?>
89-
9086
<?php if ($context->hasStudentInformation()): ?>
9187
<div class="sis-title">
9288
<?php if (isset($context->unlSISClassLevel)): ?>

0 commit comments

Comments
 (0)