wiG, ddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z ddl mZmZddlZddlmZddlmZddlmZej.eZhdZd d iZd Zej:d d d ZdZej@dejBZ"de#de$fdZ%de#de$fdZ&de#de#fdZ'de#de#fdZ(de#de e#fdZ)de#de#de e#fdZ*de de#de e#fdZ+de#de#fdZ,de#de e-e#ffd Z.d!d"d#e#ddfd$Z/de e#fd%Z0y)&N)Iterable)AnyOptionalTuple)urljoinurlparse) transaction)timezone) upload_to_ovh> image/gif image/png image/jpeg image/webpz image/jpgri@g4@g@)connectread)zno-photono_photo placeholderz/og_image_main.jpgz /static-gh/ brak_zdjeciazlogo.jpgz/gfx/logotypy/zM(?Phttps?://[^\s"\'<>]+?\.(?:jpg|jpeg|png|webp|gif))(?:[?#][^\s"\'<>]*)?ureturnc\|xsdjtfdtDS)Nc3&K|]}|v yw)N).0phlus ,/var/www/extractly/image_agregator/images.py z"_is_placeholder..#s9"8BrRx"8s)loweranyPLACEHOLDER_SUBSTRINGS)rrs @r_is_placeholderr#!s% 'r B 9"89 99sc|xsdj}|jdxr|jdxs$|jdxr|jdS)Nr[]{})strip startswithendswith)r%s r_looks_like_jsonr.%sJ bA LL  1!**S/ ]q||C7H7\QZZX[_]r$c|jjdr'|jjdsd|ddzS|jjdr'|jjdsd|ddzS|S)Nzhttps:/https://zhttp:/http://)r r,rs r_fix_scheme_slashesr5)swwwyI&qwwy/C/CJ/OAabE!!wwyH%aggi.B.B9.M1QR5  Hr$cd|xsdjjdjS)Nrz'")r+r4s r _strip_quotesr71s' G??  " "5 ) / / 11r$c\tj|xsd}|r|jdSy)uv Jeśli w stringu jest „page URL” + cytowany URL obrazka, wyciągnij pierwszy dopasowany URL obrazka. rurlN) IMG_URL_REsearchgroup)r%ms r_extract_img_url_from_textr>5s, !'r"Awwu~ r$rawbase_urlc:|syt|}t|}|xs|}t|}|j}|j ds|j drt |rdS|S|j drt |xsd|}t |rdS|Sy)u Zwraca poprawny absolutny URL obrazka dla „dziwnych” stringów: - usuwa cudzysłowy, - naprawia https:/ -> https://, - wycina URL obrazka z tekstu, jeśli są śmieci dookoła, - stosuje urljoin TYLKO dla ścieżek względnych. Nr2r0/r)r7r>r5r r,r#r)r?r@r% extractedlowabsus r_normalize_candidate_strrF?s cA*1-IQA AA '')C ~~i CNN:$>&q)t0q0 ||Cx~2q)&t,t6$6 r$cV|syt|trC|j}|syt|r t j |}nt||}|St|tr6dD]1}|j|}|stt||}|s/|cSt|trt|tttfs|D]}t|trV|jrFt|r) t j |}t||}|r|cS t||}|r|cSt|tszdD]3}|j|}|stt||}|s/|ccSy#t $rt||}|cYSwxYw#t $rYwxYw)u" Zwraca absolutny URL pierwszego sensownego obrazka. Obsługuje: - str (również „json w stringu”, np. '["https://..."]' albo '{"url":"..."}'), - dict: klucze url/src/href, - list/iterable: stringi lub dicty z ww. kluczami. Placeholdery są pomijane. N)r9srchref) isinstancestrr+r.jsonloads ExceptionrFdictget CollIterablebytes_first_image_url)r?r@r%candkvitemjs rrSrS_s #s IIK A  jjm ,Ax8DK#t'A A/AAK (#|$Zc5$=O-PD$$#D) JJt,/8<#'K 0h?K$%/A A7AI#'K 0%2 Y /8<  :%s#E?9$F?FF F('F(ctc tj|xsdjj|xsdjj}|dk(ry|dk(ry|dk(ry|dk(ry t j |xsd }|d vrdS|S) Nrr.jpgr .pngr.webpr .gifz.bin)z.jpe)CONTENT_TYPE_FIXrPr+r mimetypesguess_extension)rYexts r_ext_from_content_typercs   rxR..066828:J:J:L:R:R:T UB \ [ \ [  # #B ' 16CI%6.3.r$r9cNt|}|jr(|jr|jd|jdnd}ddd}|r||d<tjt d|5}|j d |5}|j|jjd d xsd jd d jj}tj||}|tvr|jxsd j}|j!drd}nN|j!drd}n:|j!drd}n&|j!drd}nt#d|xsdt%j&}|j)D]@} | s|j+| |j-t.kDs/t#dt.d|j1|fcdddcdddS#1swYnxYw dddy#1swYyxYw)Nz://rBz_Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36zFimage/avif,image/webp,image/apng,image/svg+xml,image/*;q=0.8,*/*;q=0.5)z User-AgentAcceptRefererT)timeoutfollow_redirectsheadersGETz content-typer;r)r[z.jpegrr\r r]rr^r zUnsupported content-type: unknownzImage too large > z bytes)rschemenetlochttpxClient HTTP_TIMEOUTstreamraise_for_statusrirPsplitr+r r_ ALLOWED_MIMEpathr- ValueErrorioBytesIO iter_byteswritetell MAX_BYTESgetvalue) r9parsedrefererriclientresprYrvbufchunks r_safe_downloadrs c]F7=}}s6==/3\`GwZG$  lT7 SW] ]]5# &$  ! ! #,,"">26<"CCCHKQQSYY[B!%%b"-B% )r002==!23%B]]6*$B]]7+%B]]6*$B$'A"/ AR%STT**,C* % 88: )$'9)F%KLL +<<>2%7' & T S & & & T S Ss+*H=EH %H2 HH HH$rnoterc f d|_t|drd|_t|drbt|dxsi}|j dg|r7|dj t jj|d||_ dg}t|dr|j dt|dr|j d|j|tjdt|d d t|d d y #t$r }tjd |Yd }~y d }~wwxYw) u Oznacz ogłoszenie do sprawdzenia stanu (status-only): - check_active=True -> trafi do get_flagged_pages(...) - check_active_from_image=True -> wiemy, że powodem była awaria obrazka Nic nie zmieniamy w is_active tutaj – zajmie się tym checker. Tcheck_active_from_imagemetaimage_failures)tsr check_active update_fieldsz5Flagged for status-check (image failure) id=%s url=%sidNr9z335tD!HM(( 86 7  !: ; 8V $   ( M 2 KWU]_ceiMjlst|DFJmK L ZUWXYYZsDD D0D++D0c t|ddxsd}t|dd}t||}|s/tjdt|ddt |d y t |\}}tj|jdd }t|}|jjj} t|dd } t!j"j%d} d| d| d| d|| } t'|| |d} | s0tj)d| t|ddt |d yt+j,5| |_|j1dgdddtjdt|dd| | S#t $rP}tjd |t|dd|t |d t|j Yd}~yd}~wwxYw#t $rR}tj)d| t|dd|dt |dt|j Yd}~yd}~wwxYw#1swYxYw)uf Pobiera pierwszy obrazek z instance.original_image_urls, wysyła do OVH i zapisuje **string** (publiczny URL) w polu `images`. Jeśli nie uda się znaleźć/pobrać/wysłać zdjęcia: - oznaczamy ogłoszenie do sprawdzenia: check_active=True, check_active_from_image=True - NIE zmieniamy od razu is_active – zrobi to checker. r9roriginal_image_urlsN)r@z&No image candidates for instance id=%srno_image_candidatesrz(Download image failed for %s (id=%s): %szdownload_error:rlz%Y/%m/%dzimages/rBz#public, max-age=31536000, immutable) content_type cache_controlz+Upload to OVH failed for key=%s (id=%s): %sT)exc_infoz upload_error:z0OVH upload returned empty URL for key=%s (id=%s)upload_empty_urlimagesrz!Stored main image for id=%s -> %s)rrSrrrrrNrtype__name__hashlibsha256 hexdigestrc __class__r r rstrftimer errorr atomicrr)rr@ originals first_urldatarYrsharbclsobj_idymd object_key public_urls rstore_main_imagersGx+1rH"7>I X>I  [\x.CD!),b ..  ( ( *3B /C  $C    % % + + -C XtY /F ,,. ! !* -C3%q#auSE:J "  ?    GU\]egkmqUrsx.@A    $ XJ /  KK3WXtT5RT^_ G A9gV^`dfjNkmnoxQ@P@P?Q.RS&  BJPWX`bfhlPmop{ Ax d1g>N>N=O.PQ  s>FG:I G7'AG22G7: IAIII!)1rerLrxrr`loggingcollections.abcrrQtypingrrr urllib.parserrro django.dbr django.utilsr cloud_storage.servicesr getLoggerrrrur_r}Timeoutrqr"compile IGNORECASEr:rKboolr#r.r5r7r>rFrSrcrRrrrrr$rrs 4''* !!0   8 $F .  u}}T3T: RZZTMM :s:t:^^^ 3 3 2S2S2#(3-##@@#@@#@F /s /s /&&&&eSj 1&&X57!Zc!Z4!ZH6(3-6r$