hdZddlmZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl mZejeZddZdd Zdd Zdd Zd d d ddZdd ddZy)a Utilities for copying (transferring) fields from NetworkMonitoredPage into the parser's extracted dict, based on a mapping stored on ManualDataSource. Config lives on ManualDataSource as JSON in field "transferred" (preferred) or legacy "trasferred". Example: { "transferred": { "listing_url": "url", "source_id": "source_id", "source_name": "source.title", "offer_kind": "offer_type", "estate_kind": "estate_type", "inactive_reason": "meta.inactive_reason" } } Integration (in parse_manual_data): from manual_agregator.parser.transferred import apply_transferred, compute_stats # ... after selectors parsing + post-rules: transferred_keys = apply_transferred(page, manual_config, extracted, skip_if_present=True) # when computing stats, exclude transferred keys so stats reflect only selector-extracted fields: extracted["_stats"] = compute_stats(selectors, extracted, selected_type, extra_exclude=set(transferred_keys)) ) annotationsN)datetime)AnyDictIterableListTuple)value_is_emptyc|siSt|tr|St|tr* tj|}t|tr|SiSiS#t $rt jdicYSwxYw)z^Return dict from raw if it's already a dict; if it's a JSON string try to parse; otherwise {}.z9transferred: failed to json-load string mapping; ignoring) isinstancedictstrjsonloads Exceptionloggerdebug)rawparseds 9/var/www/extractly/manual_agregator/parser/transferred.py_parse_json_if_strr-ss  #t #s ZZ_F'56 =2 = I  LLT UI s&AA A65A6cXt|ddxst|ddxsi}t|S)zy Prefer manual_config.transferred, fall back to legacy 'trasferred'. Always returns a dict (possibly empty). transferredN trasferred)getattrr) manual_configrs rget_transferred_mapr=s3 - 5 iP\^b9c igiC c ""c|sy|}t|jdD]5}|yt|tr|j |})t ||d}7|S)z Dot-path getter. Works with object attributes and dict keys. Example: deep_get(page, "source.title") or deep_get(page, "meta.inactive_reason") N.)rsplitr rgetr)objpathcurparts rdeep_getr(GsZ CD $ ; c4 ''$-C#tT*C % Jrc~t|tjr t|St|tr|j St|t r-|jDcic]\}}|t|c}}St|ttfr|Dcgc] }t|c}S|Scc}}wcc}w)zR Make common Django/py types JSON-serializable for json.dumps(extracted). ) r uuidUUIDrr isoformatritemsto_serializablelisttuple)valkvxs rr.r.Ys#tyy!3x#x }}#t25))+>+$!Q?1%%+>>#e}%,/0Cq"C00 J?0s )B4B:TF)skip_if_present overwritecJt|}|sgSg}|jD]W\}} t||} | |s!|r||vrt |j |s9t | ||<|j|Y|rtjd||S#t$rYwxYw)at Copy fields from NetworkMonitoredPage `page` into `extracted` according to mapping defined on ManualDataSource.transferred (or legacy trasferred). Args: page: NetworkMonitoredPage instance. manual_config: ManualDataSource instance. extracted: dict to update. skip_if_present (default True): do NOT overwrite keys that already have a non-empty value in `extracted`. overwrite (default False): hard-overwrite regardless of existing value (wins over skip_if_present). Returns: List of destination keys that were written (so the caller can exclude from stats). ztransferred: wrote keys %s) rr-r(rr r#r.appendrr) pager extractedr5r6mappingwrittendestsrc_pathr1s rapply_transferredr?hs,"-0G  G!--/h 4*C ; _y  d8K)L)#. $t!*$ 17; N%   s B B"!B") extra_excludecdh}|r|t|z}t|xsijDcgc] }||vs| c}}|jDcgc]\}}||vs t |r|}}}|xsijDcgc]$}||vst |j |s#|&} }t|||| |xsddScc}wcc}}wcc}w)z Compute stats for extracted fields, excluding internal keys and optionally transferred keys. Use this instead of ad-hoc stats if you want transferred keys excluded. _statsdefault) found_fields total_fields found_keys missing_keys type_used)setlenkeysr-r r#) selectorsr: selected_typer@excluder2rEr3rFrGs r compute_statsrOsjG3}%%IO#9#9#;P#;aq?O#;PQL ) 1` 11Qg5En]^N_! 1J` )R557q7!1G;KP^_h_l_lmn_oPpA7LqJ$ $"/i   Q`qs. C C  C$ C0C CC2C)returnr)r$rr%rrPr)r1rrPr)r:Dict[str, Any]r5boolr6rRrPz List[str]) rLrQr:rQrMz str | Noner@zIterable[str] | NonerPrQ)__doc__ __future__rrloggingr*rtypingrrrrr utilsr getLogger__name__rrrr(r.r?rOrrr[s8#  33!   8 $  #$ (! 00  0  00p+/  (   r