Exporting Raiser's Edge for CiviCRM » History » Version 61
Jon Goldberg, 03/03/2016 04:04 PM
1 | 44 | Jon Goldberg | {{lastupdated_at}} by {{lastupdated_by}} |
---|---|---|---|
2 | |||
3 | 24 | Jon Goldberg | {{toc}} |
4 | |||
5 | 1 | Jon Goldberg | h1. Exporting Raiser's Edge for CiviCRM |
6 | |||
7 | 59 | Jon Goldberg | |
8 | 23 | Jon Goldberg | There are two basic approaches to exporting RE data. There's the built-in export tool, and there's direct SQL interaction. This document will try to cover both approaches where possible. The Export tool has a lower barrier to entry, but a) there's some data you can't export with the tool, and b) the data will be denormalized, requiring additional transformation compared to extracting normalized SQL data. |
9 | 1 | Jon Goldberg | |
10 | 47 | Jon Goldberg | Note that there's a good video on this topic by Young-Jin from Emphanos, LLC, here: http://sf2013.civicrm.org/migrating-raisers-edge-civicrm |
11 | 59 | Jon Goldberg | |
12 | h2. Do you use Pentaho Kettle? |
||
13 | |||
14 | If so, you can use my Raiser's Edge to CiviCRM transforms, available here: https://github.com/PalanteJon/civicrm_kettle_transforms |
||
15 | 47 | Jon Goldberg | |
16 | 23 | Jon Goldberg | h2. Export tool - general guide. |
17 | |||
18 | The Raiser's Edge Export tool is on the left toolbar when you first enter Raiser's Edge. |
||
19 | |||
20 | 1 | Jon Goldberg | From the tool, you will create a number of exports. When you first create an export, you'll be asked a number of questions, including Export Type (Constituent, Gift, etc.), a checkbox to include inactive records (check this), and an export file type (select CSV). |
21 | |||
22 | 23 | Jon Goldberg | For most export, select Constituent as the Export type. This is the "base table" - all records will be joined relative to it. |
23 | 3 | Jon Goldberg | |
24 | 1 | Jon Goldberg | h2. Constituent Based Exports |
25 | |||
26 | 21 | Jon Goldberg | h3. Contact Information |
27 | 1 | Jon Goldberg | |
28 | 23 | Jon Goldberg | RE differentiates between constituents and non-constituents in their system. If you create a new contact, they're a constituent - but then you might decide to add a spouse or employer record, which is NOT considered a constituent, and doesn't show up in most queries. Notably, non-constituents aren't exported when using the Export tool and your base table is "Constituent". |
29 | 1 | Jon Goldberg | |
30 | 23 | Jon Goldberg | h3. SQL |
31 | |||
32 | 1 | Jon Goldberg | If extracting directly from SQL, @SELECT * FROM RECORDS@. |
33 | 21 | Jon Goldberg | |
34 | 23 | Jon Goldberg | Note that you can extract only constituents by adding @WHERE IS_CONSTITUENT = -1@. For a Civi migration, I recommend importing all contacts. |
35 | 21 | Jon Goldberg | |
36 | 23 | Jon Goldberg | h3. Export tool (NOTE: This ONLY gets constituents). |
37 | |||
38 | 1 | Jon Goldberg | Tab 1. General: |
39 | - Include all records. |
||
40 | - Head of Household processing: Export both constituents separately. |
||
41 | - Check all of the "Include these Constitutents": Inactive, deceased, no valid address |
||
42 | |||
43 | Tab 2: Output. |
||
44 | First, expand the "Constituent Information" in the left pane, and add every field to the export. Do the export (as a CSV). |
||
45 | 6 | Jon Goldberg | |
46 | 3 | Jon Goldberg | h3. Constituent Codes |
47 | 15 | Jon Goldberg | |
48 | 19 | Jon Goldberg | In RE: Found at the bottom of the "Bio 2" tab. |
49 | In SQL: CONSTITUENT_CODES maps to "GroupContact". TABLEENTRIES stores the codes ("groups"). In my case, @SELECT * FROM [CCR_July_snapshot].[dbo].[TABLEENTRIES] WHERE [CODETABLESID] = 43@ did the trick. YMMV - see "deciphering stored procedures" below. |
||
50 | 3 | Jon Goldberg | |
51 | 1 | Jon Goldberg | Export as _one to many_, below. |
52 | 3 | Jon Goldberg | These map to "groups" in Civi - can also be mapped to "tags" if you don't need to track the begin/end date on them. |
53 | |||
54 | No need to export these fields: |
||
55 | System Record ID |
||
56 | Import ID |
||
57 | 23 | Jon Goldberg | As of Civi 4.4.6, there's no way to import Group Begin/End dates via API, you need to do it via direct SQL. |
58 | 6 | Jon Goldberg | |
59 | h3. Solicit Codes |
||
60 | |||
61 | 1 | Jon Goldberg | These can map to groups - but also may map to privacy preferences or custom fields (e.g. Email Only, Do Not Solicit) |
62 | 58 | Jon Goldberg | |
63 | SQL to extract solicit codes: |
||
64 | <pre> |
||
65 | SELECT RECORDSID AS external_identifier, LONGDESCRIPTION as solicit_code FROM CONSTITUENT_SOLICITCODES JOIN TABLEENTRIES ON SOLICIT_CODE = TABLEENTRIES.TABLEENTRIESID WHERE TABLEENTRIES.ACTIVE = -1 |
||
66 | </pre> |
||
67 | |||
68 | In my copy of RE, the CODETABLESID is 5044, so to get a list of all solicit codes, use: |
||
69 | <pre> |
||
70 | SELECT LONGDESCRIPTION, ACTIVE FROM TABLEENTRIES WHERE CODETABLESID = 5044 ORDER BY SEQUENCE; |
||
71 | </pre> |
||
72 | |||
73 | 3 | Jon Goldberg | |
74 | 20 | Jon Goldberg | h3. Addresses |
75 | |||
76 | SQL tables: ADDRESS, CONSTIT_ADDRESS |
||
77 | |||
78 | Addresses are a many-to-many relationship in RE. |
||
79 | 50 | Jon Goldberg | Not all addresses in the database are visible in RE. Addresses where the @INDICATOR@ field is 1 or 7, for instance. Make sure to look your data over and filter those out accordingly. |
80 | 20 | Jon Goldberg | |
81 | 25 | Jon Goldberg | h3. Phones/E-mail/websites |
82 | 1 | Jon Goldberg | |
83 | 55 | Jon Goldberg | RE is a child of the 90's, so a) phones are tied to addresses, not contacts, and b) e-mails and websites are a type of phone. |
84 | 25 | Jon Goldberg | |
85 | 26 | Jon Goldberg | Notes: |
86 | * You can NOT have duplicate phone types in RE, so no need to try and catch multiple "Home" numbers! |
||
87 | * Oh - except that one contact can have two home phone numbers on two different addresses. |
||
88 | * Don't forget to filter out duplicate numbers/e-mails/etc. when someone puts the same phone number on two different addresses. |
||
89 | 22 | Jon Goldberg | |
90 | This SQL gets me a useful list of phones and e-mail for further processing in Kettle: |
||
91 | <pre> |
||
92 | SELECT DISTINCT |
||
93 | CONSTITADDRESSID |
||
94 | , CONSTIT_ID |
||
95 | , PHONETYPEID |
||
96 | , CONSTIT_ADDRESS_PHONES."SEQUENCE" |
||
97 | , NUM |
||
98 | , DO_NOT_CALL |
||
99 | , TEXT_MSG |
||
100 | FROM CONSTIT_ADDRESS_PHONES |
||
101 | LEFT JOIN PHONES ON CONSTIT_ADDRESS_PHONES.PHONESID = PHONES.PHONESID |
||
102 | LEFT JOIN CONSTIT_ADDRESS ON CONSTITADDRESSID = CONSTIT_ADDRESS.ID |
||
103 | </pre> |
||
104 | |||
105 | 9 | Jon Goldberg | h3. Relationships |
106 | |||
107 | 29 | Jon Goldberg | Relevant SQL table: CONSTIT_RELATIONSHIPS |
108 | |||
109 | 9 | Jon Goldberg | Relationships are different in Civi and RE in the following significant ways: |
110 | * Relationships don't have to have a relationship type. |
||
111 | 1 | Jon Goldberg | * The A-B relationship doesn't have to have the same relationship type as B-A (e.g. if my relationship is "parent", the reciprocal relationship could be "son" or "daughter". |
112 | 29 | Jon Goldberg | * Related contacts need not have their own constituent record (though they can). If they don't have their own constituent record, they nevertheless have a record in RECORDS, they're just not a constituent. |
113 | 30 | Jon Goldberg | * There need not be a relationship type at all. This doesn't make sense, except that: |
114 | * There are hardcoded fields for IS_SPOUSE, HON_MEM_ACKNOWLEDGE, IS_HEADOFHOUSEHOLD, and SOFTCREDIT_GIFTS. |
||
115 | |||
116 | 32 | Jon Goldberg | Because relationships aren't necessarily reciprocal, I find it helpful to take my list of invalid relationships and do BOTH of the following: |
117 | * Look up the RELATIONSHIP_TYPE against the @name_b_a@ field in @civicrm_relationship_type@. |
||
118 | * Look up the RECIP_RELATIONSHIP_TYPE against both @name_a_b@ and @name_b_a@ in @civicrm_relationship_type@. |
||
119 | 9 | Jon Goldberg | |
120 | 53 | Jon Goldberg | h3. Solicitor Relationships |
121 | |||
122 | Solicitor relationships are stored in a different table. I used this SQL to extract them: |
||
123 | <pre> |
||
124 | SELECT |
||
125 | CONSTIT_ID |
||
126 | , SOLICITOR_ID |
||
127 | , TABLEENTRIES.LONGDESCRIPTION as solicitor_type |
||
128 | , AMOUNT |
||
129 | , NOTES |
||
130 | , cs."SEQUENCE" as weight |
||
131 | FROM CONSTIT_SOLICITORS cs |
||
132 | LEFT JOIN TABLEENTRIES ON cs.SOLICITOR_TYPE = TABLEENTRIES.TABLEENTRIESID |
||
133 | ORDER BY weight |
||
134 | </pre> |
||
135 | |||
136 | 10 | Jon Goldberg | h3. Attributes |
137 | |||
138 | Attributes are the RE equivalent of custom fields. However, unlike custom fields, they can also have a "date" value and a "comments" value. While this can be replicated in Civi via multi-record custom field groups, ideally the data is evaluated attribute by attribute. |
||
139 | |||
140 | 11 | Jon Goldberg | Valuable information about the setup of the attributes is available in RE from *Config > Attributes*. |
141 | 1 | Jon Goldberg | |
142 | 33 | Jon Goldberg | * The analogous field to @civicrm_custom_field@ is @AttributeTypes@. |
143 | * @AttributeTypes.CODETABLESID@ gives a lookup for the RE "option group" that contains valid options for that attribute. |
||
144 | 37 | Jon Goldberg | * All constituent attribute data is stored in the table @ConstituentAttributes@. Note that it's stored in a Key-Value Pair-style table - you'll need to do a bunch of SQL queries, or run a Kettle "Row Denormaliser" step to get this data in order. |
145 | |||
146 | 33 | Jon Goldberg | Here's my preliminary SQL to export attributes from RE: |
147 | <pre> |
||
148 | SELECT |
||
149 | 36 | Jon Goldberg | ca.PARENTID as external_identifier |
150 | , ca.ATTRIBUTETYPESID |
||
151 | , at.DESCRIPTION as Category |
||
152 | , TABLEENTRIES.LONGDESCRIPTION as Description |
||
153 | 33 | Jon Goldberg | , TEXT |
154 | , NUM |
||
155 | , DATETIME |
||
156 | , CURRENCY |
||
157 | , "BOOLEAN" |
||
158 | 35 | Jon Goldberg | , COMMENTS |
159 | 33 | Jon Goldberg | , ca.ATTRIBUTEDATE |
160 | FROM ConstituentAttributes ca |
||
161 | JOIN AttributeTypes at ON ca.ATTRIBUTETYPESID = at.ATTRIBUTETYPESID |
||
162 | 36 | Jon Goldberg | LEFT JOIN TABLEENTRIES ON ca.TABLEENTRIESID = TABLEENTRIES.TABLEENTRIESID |
163 | 33 | Jon Goldberg | </pre> |
164 | |||
165 | 34 | Jon Goldberg | *note:* In the SQL above, "PARENTID" and not "ConstitID" is the correct foreign key to link this to the contact. |
166 | |||
167 | 38 | Jon Goldberg | To get a list of option values out of RE for the attributes, use this SQL: |
168 | <pre> |
||
169 | SELECT |
||
170 | DESCRIPTION |
||
171 | , at.CODETABLESID |
||
172 | , LONGDESCRIPTION |
||
173 | FROM TABLEENTRIES te |
||
174 | LEFT JOIN AttributeTypes at ON te.CODETABLESID = at.CODETABLESID |
||
175 | ORDER BY DESCRIPTION |
||
176 | </pre> |
||
177 | |||
178 | 39 | Jon Goldberg | Attributes can be multi-record custom fields by their nature, so you have to account for that. Here's some alpha-grade SQL for sussing out which fields have multi-record custom fields: |
179 | <pre> |
||
180 | SELECT ATTRIBUTETYPESID, PARENTID, COUNT(LONGDESCRIPTION) |
||
181 | FROM ConstituentAttributes ca |
||
182 | JOIN TABLEENTRIES te ON ca.TABLEENTRIESID = te.TABLEENTRIESID |
||
183 | GROUP BY PARENTID, ATTRIBUTETYPESID |
||
184 | HAVING COUNT(LONGDESCRIPTION) > 1 |
||
185 | ORDER BY ATTRIBUTETYPESID |
||
186 | </pre> |
||
187 | |||
188 | 38 | Jon Goldberg | *note:* In Civi 4.5+, you could conceivable use "EntityRef" functionality to facilitate chained selects of OptionValue lists. That would let you create a multi-record custom field group that would very closely map how Attributes work in RE - but you'd have all the disadvantages of multi-record custom fields. |
189 | 10 | Jon Goldberg | |
190 | 49 | Jon Goldberg | h3. Salutations/addressee info |
191 | 48 | Jon Goldberg | |
192 | 49 | Jon Goldberg | RE stores contact salutations and addressee info in two places. |
193 | 48 | Jon Goldberg | |
194 | 49 | Jon Goldberg | Primary salutations/addressess are stored on the @RECORDS@ table. @PRIMARY_ADDRESSEE_ID@, @PRIMARY_ADDRESSEE@, and @PRIMARY_ADDRESSEE_EDIT@, @PRIMARY_SALUTATION_ID@, @PRIMARY_SALUTATION@, @PRIMARY_SALUTATION_EDIT@. |
195 | 1 | Jon Goldberg | |
196 | 49 | Jon Goldberg | An unlimited number of non-primary salutations can be stored in the @CONSTITUENT_SALUTATION@ table. |
197 | 48 | Jon Goldberg | |
198 | 49 | Jon Goldberg | Salutation options values are stored in the SALUTATION table, in the format "CODE1, CODE2, CODE3, etc.". Each code refers to an id in the SALUTATION_FIELDS table, which contains tokens (e.g. "First Name", "Spouse Last Name") as well as common words like "And". |
199 | |||
200 | Note that @PRIMARY_ADDRESSEE@ is more akin to @addressee_display@ in Civi, in that it stores the calculated display ID. Also note that when @PRIMARY_ADDRESSEE_EDIT@ is -1 (true), that's the equivalent of a custom addressee in Civi, and the value stored in @PRIMARY_ADDRESSEE_ID@ must be ignored. |
||
201 | 48 | Jon Goldberg | |
202 | 3 | Jon Goldberg | h3. Other constituent tables: |
203 | 5 | Jon Goldberg | |
204 | 2 | Jon Goldberg | Skip these tables: |
205 | * Spouse |
||
206 | * Gifts |
||
207 | * First Gift, Last gift, Largest Gift |
||
208 | * Actions |
||
209 | 1 | Jon Goldberg | * First Action, Last Action |
210 | * Summary Information |
||
211 | |||
212 | 41 | Jon Goldberg | h2. Contribution-related exports |
213 | |||
214 | h3. Contributions/Gifts |
||
215 | |||
216 | Contributions (in RE parlance: Gifts) are complicated beasts! |
||
217 | |||
218 | Here are some relevant database tables and their equivalent in Civi: |
||
219 | GIFT civicrm_contribution |
||
220 | GiftSplit civicrm_line_item |
||
221 | CAMPAIGN Roughly maps to Campaign. Your mapping may vary and/or include custom fields. |
||
222 | APPEAL Also roughly maps to Campaign (or Source). Your mapping may vary and/or include custom fields. |
||
223 | FUND Roughly maps to Financial Type. Your mapping may vary and/or include custom fields. |
||
224 | |||
225 | Note that gift type is hardcoded into a function called "TranslateGiftType) - so you may want to include that function in your SQL, e.g.: |
||
226 | <pre> |
||
227 | SELECT |
||
228 | gs.GiftId |
||
229 | , g.CONSTIT_ID |
||
230 | , gs.Amount |
||
231 | , g.DTE as gift_date |
||
232 | , FUND.DESCRIPTION as fund |
||
233 | , CAMPAIGN.DESCRIPTION as campaign |
||
234 | , APPEAL.DESCRIPTION as appeal |
||
235 | , g.PAYMENT_TYPE |
||
236 | , g.ACKNOWLEDGE_FLAG |
||
237 | , g.CHECK_NUMBER |
||
238 | , g.CHECK_DATE |
||
239 | , g.BATCH_NUMBER |
||
240 | , g.ANONYMOUS |
||
241 | , gst.LONGDESCRIPTION as giftsubtype |
||
242 | , g.TYPE |
||
243 | , DBO.TranslateGiftType(g.TYPE) as type2 |
||
244 | FROM GiftSplit gs |
||
245 | LEFT JOIN FUND on gs.FundId = FUND.id |
||
246 | LEFT JOIN APPEAL on gs.AppealId = APPEAL.id |
||
247 | LEFT JOIN CAMPAIGN on gs.CampaignId = CAMPAIGN.id |
||
248 | LEFT JOIN GIFT g on gs.GiftId = g.ID |
||
249 | LEFT JOIN TABLEENTRIES gst on g.GIFTSUBTYPE = gst.TABLEENTRIESID |
||
250 | </pre> |
||
251 | |||
252 | 42 | Jon Goldberg | Payment Type is also hard-coded, it seems: |
253 | 1 Cash |
||
254 | 2 Personal Check |
||
255 | 3 Business Check |
||
256 | 4 Credit Card |
||
257 | 6 Direct Debit |
||
258 | 8 Other |
||
259 | |||
260 | h3. Soft Credits |
||
261 | |||
262 | Stored in GIFTSOFTCREDIT. RE does NOT have the concept of a soft credit type - which is fine. |
||
263 | <pre> |
||
264 | SELECT |
||
265 | , GiftId |
||
266 | , ConstitId |
||
267 | , Amount |
||
268 | , 'Soft Credit' as soft_credit_type |
||
269 | FROM GiftSoftCredit |
||
270 | </pre> |
||
271 | |||
272 | 52 | Jon Goldberg | h3. Solicitor, Gift |
273 | |||
274 | (Important! Gift solicitors are different from Contact Solicitors) |
||
275 | 42 | Jon Goldberg | |
276 | I imported these as soft credits, but a different TYPE of soft credit. Here's the SQL I used to get the data out of RE: |
||
277 | <pre> |
||
278 | SELECT |
||
279 | ParentId as gift_id |
||
280 | , SolicitorId as soft_creditee_external_identifier |
||
281 | , Amount |
||
282 | , 'Solicitor' as soft_credit_type |
||
283 | FROM GiftSolicitor |
||
284 | </pre> |
||
285 | |||
286 | 41 | Jon Goldberg | h3. In Honor/Memorial Of (aka Tributes) |
287 | |||
288 | As of CiviCRM 4.5, In Honor/Memorial of is considered a form of soft credit. In RE, they're still separate, and are called Tributes. The structure is a little more complex - the table structure is Constituent <-> Tribute <-> Gift_Tribute <-> Gift. Civi is Contact <-> Soft Credit <-> Contribution. |
||
289 | |||
290 | Here is some preliminary SQL that pulls tribute data suitable for transformation and import to Civi as ContributionSoft entities. Note that CiviCRM doesn't have a concept of a "Description" but does have the concept of a PCP Note, so I'm importing the description there - in the future, I could see the argument for Civi exposing the PCP Note as a description. |
||
291 | |||
292 | <pre> |
||
293 | SELECT |
||
294 | gt.GIFT_ID |
||
295 | , gt.TRIBUTE_TYPE |
||
296 | , t.DESCRIPTION |
||
297 | , t.RECORDS_ID as tributee_extenal_identifier |
||
298 | , te.LONGDESCRIPTION as tribute_type |
||
299 | FROM GIFT_TRIBUTE gt |
||
300 | JOIN TRIBUTE t ON gt.TRIBUTE_ID = t.ID |
||
301 | LEFT JOIN TABLEENTRIES te on gt.TRIBUTE_TYPE = te.TABLEENTRIESID |
||
302 | </pre> |
||
303 | |||
304 | 51 | Jon Goldberg | h2. Actions |
305 | |||
306 | Actions fill the same purpose as Activities in CiviCRM, but are architected quite differently - in some ways better, some ways worse. I don't have as much concrete info here, but here's a decent start at extracting Actions data via SQL: |
||
307 | <pre> |
||
308 | SELECT |
||
309 | a.ADDED_BY |
||
310 | , a.AUTO_REMIND |
||
311 | , a.RECORDS_ID as external_identifier |
||
312 | , cr.RELATION_ID as action_contact_id |
||
313 | , a.DTE as activity_date_time |
||
314 | , LETTER.LONGDESCRIPTION as letter |
||
315 | , a.PRIORITY as priority_id |
||
316 | , a.REMIND_VALUE |
||
317 | , a.CATEGORY |
||
318 | , a.Completed |
||
319 | , a.COMPLETED_DATE |
||
320 | , a.FUND_ID |
||
321 | , a.FOLLOWUPTO_ID |
||
322 | , a.TRACKACTION_ID |
||
323 | , a.PhoneNumber as phone_number |
||
324 | , a.Remind_Frequency |
||
325 | , a.WORDDOCNAME |
||
326 | , a.APPEAL_ID |
||
327 | , a.APPEAL_LETTER_CODE |
||
328 | , a.OUTLOOK_EMAIL_SUBJECT |
||
329 | , STATUS.LONGDESCRIPTION as status |
||
330 | , TYPE.LONGDESCRIPTION as type |
||
331 | , LOCATION.LONGDESCRIPTION as location |
||
332 | , ActionNotepad.ActualNotes |
||
333 | , CAMPAIGN.DESCRIPTION as campaign |
||
334 | FROM ACTIONS a |
||
335 | LEFT JOIN TABLEENTRIES as STATUS ON a.STATUS = STATUS.TABLEENTRIESID |
||
336 | LEFT JOIN TABLEENTRIES as TYPE ON a.[TYPE] = [TYPE].TABLEENTRIESID |
||
337 | LEFT JOIN TABLEENTRIES as LOCATION ON a.[Location] = LOCATION.TABLEENTRIESID |
||
338 | LEFT JOIN TABLEENTRIES as LETTER on a.[LETTER_CODE] = LETTER.TABLEENTRIESID |
||
339 | 56 | Jon Goldberg | LEFT JOIN ActionNotepad ON a.ID = ActionNotepad.ParentId |
340 | 51 | Jon Goldberg | LEFT JOIN CAMPAIGN on a.CAMPAIGN_ID = CAMPAIGN.id |
341 | LEFT JOIN CONSTIT_RELATIONSHIPS cr on a.CONTACT_ID = cr.ID |
||
342 | </pre> |
||
343 | |||
344 | 1 | Jon Goldberg | "Category" and "Action type" both roughly map to "Activity Type". Same for "status" and "COMPLETED" and "COMPLETED_DATE" mapping to "activity_status". RE lets you designate a related Campaign, Fund and Proposal; out of the box, Civi only supports Campaign. The auto-reminder is more flexible than you can get with scheduled reminders in Civi without getting very complicated. "Solicitors" can't be mapped to a contact reference lookup, because more than one can be stored. |
345 | 56 | Jon Goldberg | |
346 | *Note:* The SQL above presumes only one note per action. If you have multiple notes per action, the action will be represented with multiple records, one per associated note. I'll try to provide SQL for extracting the notes separately at a later date. |
||
347 | 51 | Jon Goldberg | |
348 | 57 | Jon Goldberg | h2. Action Notes |
349 | |||
350 | Action Notes are stored in their own table. This maps to "Details" on a Civi activity, but you can log multiple notes per action in RE. Here's the SQL I used to extract them in preparation: |
||
351 | <pre> |
||
352 | SELECT |
||
353 | NotesID |
||
354 | , Title |
||
355 | , Description |
||
356 | , Author |
||
357 | , ActualNotes |
||
358 | , ParentId |
||
359 | , NotepadDate |
||
360 | , TABLEENTRIES.LONGDESCRIPTION as Type |
||
361 | FROM ActionNotepad |
||
362 | LEFT JOIN TABLEENTRIES ON ActionNotepad.NoteTypeId = TABLEENTRIES.TABLEENTRIESID |
||
363 | ORDER BY ParentId, ActionNotepad."SEQUENCE" |
||
364 | </pre> |
||
365 | |||
366 | 54 | Jon Goldberg | h2. Events |
367 | |||
368 | Events are stored fairly similarly to CiviCRM, except: |
||
369 | * locations are stored on the event record itself (which I'm not dealing with). |
||
370 | * There's fields for storing data about classes. I haven't delved into this - I suspect that this may tie into recurring events. |
||
371 | * "Event Category" and "Event Type" might both map to Civi's "Event Type". This isn't the case for me. |
||
372 | * Events need not have begin/end dates. While this is technically true for Civi, you're buying yourself a whole lot of trouble. I'm pulling in "DATE_ADDED" to substitute in for START_DATE where none exists. |
||
373 | |||
374 | Here's some SQL to pull in the most relevant data: |
||
375 | <pre> |
||
376 | SELECT |
||
377 | se.CAPACITY |
||
378 | , se.END_DATE |
||
379 | , se.ID |
||
380 | , se.NAME |
||
381 | , se.START_DATE |
||
382 | , se.DATE_ADDED |
||
383 | , te.LONGDESCRIPTION as activity_type |
||
384 | , se.INACTIVE |
||
385 | , se.DISPLAYONCALENDAR |
||
386 | , CAMPAIGN.DESCRIPTION as campaign |
||
387 | , se.DESCRIPTION |
||
388 | FROM SPECIAL_EVENT se |
||
389 | LEFT JOIN CAMPAIGN on se.CAMPAIGN_ID = CAMPAIGN.id |
||
390 | LEFT JOIN TABLEENTRIES te ON se.TYPEID = te.TABLEENTRIESID |
||
391 | </pre> |
||
392 | |||
393 | 45 | Jon Goldberg | h2. Notes |
394 | |||
395 | RE notes (stored in the "ConstituentNotepad" table) can store quite a bit of data that Civi notes can not. They can store formatting (but with proprietary format tags, not HTML), inline photos, etc, and contain fields for date of note (separate from "Date Added" and "Date Changed"), the type of note, etc. Fortunately, they store plain-text versions of formatted notes in their own field. "Notes" is formatted; "ActualNotes" is plain text (except, well, where it isn't). |
||
396 | |||
397 | I've resolved this by removing notes over a certain length (above 15K and I assume you're a photo) and concatenating the fields I want to keep (e.g. Note Type and Description) with the ActualNotes field. |
||
398 | |||
399 | 60 | Jon Goldberg | It may be possible to export the photos in the Notes using the techniques described below under "Media". |
400 | 45 | Jon Goldberg | |
401 | 46 | Jon Goldberg | Here's the SQL I'm currently using to extract notes before doing transforms in Kettle: |
402 | <pre> |
||
403 | SELECT |
||
404 | Title as subject |
||
405 | , Description |
||
406 | , Author |
||
407 | , ActualNotes |
||
408 | , ParentId |
||
409 | , cn.DateChanged |
||
410 | , LONGDESCRIPTION as NoteType |
||
411 | FROM ConstituentNotepad cn |
||
412 | LEFT JOIN TABLEENTRIES ON NoteTypeId = TABLEENTRIESID |
||
413 | </pre> |
||
414 | |||
415 | 61 | Jon Goldberg | h2. Media |
416 | |||
417 | The files stored on the "Media" tab are held in the [dbo].[MEDIA] table in MS SQL. Assuming embedded and not linked data, the files are stored in the MS Access OLE format. It's relatively difficult to extract data from the OLE wrapper, though searching for @extract access ole@ on any search engine will give you lots of options in a variety of languages. Blackbaud even has code to do it "here":https://kb.blackbaud.com/articles/Article/58559, if you feel like using VBA. |
||
418 | |||
419 | I opted to use a commercial software package from Yohz Software called "SQL Image Viewer":http://www.yohz.com/siv_details.htm. You can enter the command @SELECT OBJECT FROM [dbo].[MEDIA], press "Execute Query", and then press "Export" when it's done. Note that this will take a VERY long time if you have even a couple hundred megs of media! |
||
420 | |||
421 | COMING MARCH 2016: How to import these files into CiviCRM as attachments to activities. |
||
422 | |||
423 | 43 | Jon Goldberg | h2. And More |
424 | |||
425 | 7 | Jon Goldberg | h3. Tables that Civi doesn't have a direct counterpart for |
426 | 5 | Jon Goldberg | |
427 | 3 | Jon Goldberg | * Aliases (stores Maiden Name and d/b/a - unsure how to import into Civi just yet) |
428 | 7 | Jon Goldberg | * Solicitor Goals - Can be found on an RE contact record on "Bio 1" tab by clicking "Details" next to "Is a Solicitor" checkbox. Don't know how to use them. |
429 | 2 | Jon Goldberg | |
430 | |||
431 | Open each CSV file in Excel or similar. Sort each field by ascending AND descending to see if any data is stored in that field. If every record has no data or the same data, delete it - it's not being tracked in the current system. If you see only one or two records with a particular field, they're also probably fine to go, but check with the client first. |
||
432 | |||
433 | 1 | Jon Goldberg | |
434 | Next, strip out all of the constituent information except for primary/foreign keys. I like to keep in First/Middle/Last name just for human readability though. So leave in those three fields, plus any field with the word "ID" in it. This is your base constituent info, and will be in every other export you do. |
||
435 | |||
436 | Now comes the fun part! Export each table, one at a time, by adding those fields to an export that already includes the base constituent info. |
||
437 | |||
438 | For one-to-many relationships, the system will ask you how many instances of the information to export. I default to 12, then look over the data to see how many are actually used, then re-export with a higher or lower number. |
||
439 | |||
440 | I also remove records that don't contain the relevant data. For instance, when exporting Solicit Codes, I sort by the first Solicit Code. Then I scroll down past the folks that have Solicit Codes to those who have none, and delete the rows for folks who have none. |
||
441 | |||
442 | Note that for simplicity's sake, RE contains many views of the tables that, if you export them all, you'll have redundant data. There's no need to export "First Gift", "Last Gift", or "Largest Gift" - simply export all gifts. Likewise for "Preferred Address". |
||
443 | |||
444 | When exporting one-to-many tables that themselves contain one-to-many tables (e.g. Addresses contains Phones), do NOT select 12 of each! That means you're exporting 144 phone numbers per record. First determine the maximum number of addresses being tracked, re-export with that number, THEN export with phone numbers. Also, it's reasonable to export with 5 phone numbers per address. |
||
445 | |||
446 | NOTE: Letters sent is incomplete, there's more than 12 letters to some folks! |
||
447 | |||
448 | GIFTS is related to constituent on the last column (Constituent System Record ID) |
||
449 | 8 | Jon Goldberg | |
450 | 13 | Jon Goldberg | h3. Code Tables/Option Groups/Option Values |
451 | |||
452 | 17 | Jon Goldberg | If you're extracting data from the SQL back-end, you'll see that the RE equivalent to Civi option groups is "code tables". There's two functions that handle lookups: dbo.GetTableEntryDescription and dbo.GetTableEntryDescSlim. To determine where the data is being accessed by the function, see "Deciphering MS SQL", below. Use the "lTableNumber" passed to those functions and you'll find your data in dbo.CODETABLES (comparable to civicrm_option_group), dbo.CODETABLEMAP and dbo.TABLEENTRIES (comparable to civicrm_option_value). |
453 | |||
454 | h2. Deciphering MS SQL |
||
455 | |||
456 | SQL Server Profiler is a tool that lets you spy on SQL statements passed to MS SQL, which is good for determining where certain data lives. However, RE depends on functions and stored procedures, so sometimes the SQL won't tell you exactly where to look. |
||
457 | |||
458 | h3. Looking Up Functions |
||
459 | |||
460 | These are embedded in SQL and have a nomenclature like: dbo.GetTableEntryDescSlim. Find them in SQL Server Management Studio: database > Programmability > Functions > Scalar-valued Functions. |
||
461 | |||
462 | h3. Looking Up Stored Procedures |
||
463 | |||
464 | 18 | Jon Goldberg | If, in the profiler, taking a certain action shows a command like this: |
465 | 17 | Jon Goldberg | These have a syntax like: |
466 | 1 | Jon Goldberg | <pre> |
467 | 18 | Jon Goldberg | exec sp_execute 48,43,'Acknowledgee' |
468 | 1 | Jon Goldberg | </pre> |
469 | |||
470 | 18 | Jon Goldberg | You're dealing with a stored procedure. You need to find the corresponding @exec sp_prepexec@ command (in this case, the one with a 48). In this case, it looks like: |
471 | <pre> |
||
472 | declare @p1 int |
||
473 | set @p1=48 |
||
474 | exec sp_prepexec @p1 output,N'@P1 int,@P2 varchar(255)',N'SELECT Top 1 TABLEENTRIESID FROM DBO.TABLEENTRIES WHERE CODETABLESID = @P1 AND LONGDESCRIPTION = @P2 ',43,'Acknowledgee' |
||
475 | select @p1 |
||
476 | </pre> |
||
477 | |||
478 | 40 | Jon Goldberg | Note that there's a tool called "SQL Hunting Dog", a free plug-in for SQL Server Management Studio, which makes locating stored procedures, etc. easier. |
479 | 17 | Jon Goldberg | |
480 | 13 | Jon Goldberg | |
481 | 14 | Jon Goldberg | h3. Addressee/Postal Greeting/E-mail greeting |
482 | |||
483 | RE has a much wider variety of greeting formats out-of-the-box. The "spouse ID" is stored on the record to enable quick lookups of addressee greetings that include the spouse. |
||
484 | |||
485 | 8 | Jon Goldberg | See also: |
486 | http://support.littlegreenlight.com/kb/migration/migrating-from-the-raisers-edge-to-lgl |
||
487 | 14 | Jon Goldberg | |
488 | h3. Things I see that RE does better than Civi: |
||
489 | |||
490 | * Better greetings/salutations UI out of the box. In Civi, you must in-line edit the greetings, then press "Edit" next to the greetings, and even then you only see the tokens you'll use. RE lets you edit with no clicks, and parses the tokens for you. |
||
491 | * The equivalent of option values are stored with their id, not their value. This isn't a big deal, but it DOES make data transformation easier in RE, and I suspect it makes their equivalent of pseudoconstant code easier to read. |
||
492 | 28 | Jon Goldberg | * There's a lot more data stored in many-to-many tables. For instance, job titles are stored in the relationship tab, reflecting the fact that someone can have more than one job. |