Compare commits

695 Commits

Author SHA1 Message Date
王文艳 2d266a30c2 feat(office): auto-calculate mom and yoy for city, district, and office monthly stats 2026-10-19 11:54:35 +08:00
王文艳 0d65af4be7 fix(office): filter abnormal rent sample prices and fix city variable shadowing in base_price 2026-10-19 10:46:15 +08:00
王文艳 fb8f6400be fix(office): 延迟导入业务模块并补充 .airflowignore 及 __main__ 保护以解决 DagBag 导入超时 2026-10-19 09:38:44 +08:00
王文艳 0b7caf6a9f fix(office): 将 fill_base_price 数据库连接移至函数内部以避免 DagBag 导入超时 2026-10-19 09:30:54 +08:00
王文艳 92802d2d50 feat(api): njcb_village_statement 调整接收人员 guyzh@njcb.com.cn 及抄送人员 2026-10-08 09:53:15 +08:00
王文艳 e79e154547 feat(poi): 缺少经纬度区域增加高德地理编码获取及人工审核列 2026-09-20 08:25:08 +08:00
王文艳 8f502b42b7 feat(poi): 将缺少有效经纬度的区域单独导出到Excel的'缺少经纬度区域'Sheet 2026-09-16 10:15:17 +08:00
王文艳 3e927f2c6d feat(poi): 过滤跳过台湾省行政区划 2026-09-16 10:09:57 +08:00
王文艳 21be81690f feat(poi): 过滤区级区域(以'区'结尾),高德QPS超限(10021)时增加0.2秒重试等待 2026-09-16 10:02:26 +08:00
王文艳 3a301eea2e feat(poi): 支持通过dag_run.conf或Airflow UI params自定义高德API Key 2026-09-16 09:56:42 +08:00
王文艳 cccafbe651 feat: 对公涉农仅保留判定为农的记录,且两个定时任务邮件均抄送至daijinshui@fangjia.com 2026-09-16 09:43:54 +08:00
王文艳 6264d35a54 refactor(poi): 将县政府乡镇街道核验任务整合为单一DAG文件,避免拆分导致的邮件发送环境问题 2026-09-16 09:39:03 +08:00
王文艳 9134b28713 feat(poi): 添加县政府所在乡镇街道核验排查脚本及月度定时任务 2026-09-16 09:36:13 +08:00
王文艳 5711fb84a8 fix(api): 修复res_map嵌套result取值,区分输入输出前缀,调整邮件称呼与文件名 2026-09-16 09:24:27 +08:00
王文艳 a5e0ae9a0d fix(api): 切换为PythonOperator以支持原生Airflow环境执行邮件发送 2026-09-16 09:01:17 +08:00
王文艳 df9dd81487 refactor(api): 拆分南京银行涉农API对账单DAG与执行脚本为BashOperator模式 2026-09-16 08:50:41 +08:00
王文艳 2b72b6597a feat(api): 添加南京银行涉农API月度对账单定时任务 2026-09-16 08:40:02 +08:00
王文艳 dba583f1c4 fix(community): filter non-positive prices when syncing ES prices to MongoDB 2026-08-28 10:37:58 +08:00
王文艳 b2f92c12f2 feat: 在 sell/base_price.py 写字楼出售重算路径中追加自动计算 mom/yoy 环比同比指标,避免重算后指标丢失 2026-08-17 13:14:27 +08:00
王文艳 c4556a9271 feat: 在 base_price.py 的单写字楼重算路径中追加自动计算 mom/yoy 环比同比指标的后置处理,避免重算后指标丢失 2026-08-17 12:56:46 +08:00
王文艳 3bdbc5ca8c 版本修改 2026-08-17 12:32:51 +08:00
王文艳 9800abcbc2 版本修改 2026-08-17 12:31:06 +08:00
王文艳 5603858a93 fix: 在 bd2gaode.py 中增加 hasattr 校验,兼容低版本 elasticsearch 客户端,防止 options() 报错 2026-08-17 12:29:33 +08:00
王文艳 bf2923bdc4 chore: 移除临时的数据库维护脚本 clean_dags_db.py,防止 Airflow 3.0 沙箱限制报错 2026-08-17 12:28:29 +08:00
王文艳 fff45fbd37 refactor: 全量升级 31 个 DAG 的 BashOperator 导入路径,消除 Airflow 3.0 的废弃警告 2026-08-17 10:02:21 +08:00
王文艳 726ac630f5 refactor: 升级 BashOperator 的导入路径以消除 Airflow 3.0 的废弃警告 2026-08-17 10:01:15 +08:00
王文艳 5110fee974 fix: 移除 ES 206 匿名集群的账号密码认证以解决 401 报错 2026-08-17 09:58:17 +08:00
王文艳 77186ee35c 版本修改 2026-08-17 09:31:32 +08:00
王文艳 ac01d98e08 版本修改 2026-08-17 09:28:50 +08:00
王文艳 d04bdbd464 feat: ClickHouse 彻底下线并平替为 MongoDB tmp 库,新增 ES 206 均价至 MongoDB 85 月度同步任务 2026-08-17 09:08:45 +08:00
王文艳 6dae2f895c fix: add immediate elasticsearch compatibility hotfix to del_comm.py 2026-08-16 22:35:53 +08:00
王文艳 3f09013747 版本修改 2026-08-16 22:25:09 +08:00
王文艳 4721f23827 版本修改 2026-07-22 16:29:13 +08:00
王文艳 e36d16cbdd 版本修改 2026-07-22 16:19:17 +08:00
王文艳 b7534064aa 版本修改 2026-07-22 16:13:44 +08:00
王文艳 a11383312e 版本修改 2026-07-22 16:06:26 +08:00
王文艳 eddb99892a 版本修改 2026-07-22 16:02:31 +08:00
王文艳 aee4c00628 版本修改 2026-07-22 15:42:34 +08:00
王文艳 bfcdb591d3 版本修改 2026-07-22 15:17:02 +08:00
王文艳 9467a62152 版本修改 2026-07-22 11:41:49 +08:00
王文艳 327b36e5a0 版本修改 2026-07-22 11:10:14 +08:00
王文艳 aed37e0759 Merge remote-tracking branch 'origin/master' 2026-07-22 09:33:22 +08:00
王文艳 ea0a812f09 版本修改 2026-07-22 09:31:16 +08:00
muliangshu afa57f7261 Merge remote-tracking branch 'origin/master' 2026-03-26 16:01:33 +08:00
muliangshu 8f5b35679b stock_new更新 2026-03-26 16:00:33 +08:00
admin 009e3559c9 Update stock.py 2026-03-04 14:53:45 +08:00
muliangshu 9faf6baa72 Merge remote-tracking branch 'origin/master' 2026-01-25 20:48:23 +08:00
muliangshu 6d88b5e31d stock_new更新 2026-01-25 20:47:42 +08:00
admin 740353e0a7 移除无用的工具包 2025-12-29 15:18:14 +08:00
admin 8fd76993fc 移除无用的工具包 2025-12-29 15:16:42 +08:00
admin 4b9c8f6d19 移除无用的工具包 2025-12-29 15:01:39 +08:00
admin ab7d061946 添加区域财力评级 2025-12-29 12:00:33 +08:00
admin 6289e30cfe 修改股票表结构 2025-12-24 10:00:41 +08:00
muliangshu 21f6354658 stock_new更新 2025-12-24 08:18:19 +08:00
admin 70224430b9 修改股票表结构 2025-12-23 09:09:44 +08:00
admin 54012fc635 添加API启动支持 2025-12-11 16:02:17 +08:00
admin 94f509725a 添加API启动支持 2025-11-12 21:58:47 +08:00
admin 2e49c448d4 fix bug 2025-11-11 18:40:59 +08:00
admin 2de5f549ed fix bug 2025-11-11 17:43:35 +08:00
admin 37ecd3095d 修改售价对api的支持 2025-11-11 17:41:03 +08:00
admin 06551bff10 修改bug 2025-11-11 17:23:32 +08:00
admin 80d960dd4c 添加API启动支持 2025-11-11 17:08:22 +08:00
admin eec268365b 更新定时信息 2025-10-28 14:41:56 +08:00
admin d4fdc75146 修改数据类型 2025-10-28 13:23:16 +08:00
admin 7ea3f83374 修改数据类型 2025-10-28 13:08:56 +08:00
admin 1b030c9522 优化日期类型 2025-10-28 11:00:02 +08:00
admin 2f6a40d64a 修改bug 2025-10-28 09:28:14 +08:00
admin e6922e4c6b fix bug 2025-10-27 21:50:28 +08:00
admin 617db91a11 修改bug 2025-10-14 08:36:06 +08:00
admin f42e32065e 区域评级异常修复 2025-10-14 08:22:52 +08:00
admin 641356bd07 修改楼盘评级bug 2025-10-13 20:08:01 +08:00
admin 02a1c27c0c fix bug 2025-08-20 14:33:34 +08:00
admin c1168b02a8 fix bug 2025-08-20 14:10:55 +08:00
admin ef5601e0e0 fix bug 2025-08-20 14:04:56 +08:00
admin 203920ad6c fix bug 2025-08-20 14:00:31 +08:00
admin ece4633ab1 fix bug 2025-08-20 12:32:28 +08:00
admin e5d39afc20 fix bug 2025-08-20 12:28:59 +08:00
admin fde96a1886 fix bug 2025-08-20 12:26:47 +08:00
admin 27a8f78f62 fix bug 2025-08-20 12:23:09 +08:00
admin 50597d9a5a 添加日志 2025-08-20 12:20:14 +08:00
admin 7f286359e1 添加区域财力评级 2025-07-06 11:21:38 +08:00
admin e31aaf6794 添加肉 2025-07-05 22:35:42 +08:00
admin 93c5d9501b 恢复加工数据到原始数据 2025-07-03 18:27:43 +08:00
admin 125937a96c 恢复加工数据到原始数据 2025-07-03 18:24:53 +08:00
admin 5ef8c21a3b 恢复加工数据到原始数据 2025-07-03 18:10:11 +08:00
admin 5b8fe6947c Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2025-07-03 16:03:53 +08:00
admin 047b826691 优化楼盘评级算法 2025-07-03 16:03:07 +08:00
沈健 f892dcee0a 优化 2025-06-20 15:07:09 +08:00
沈健 7880ccf802 优化 2025-06-20 14:57:40 +08:00
沈健 a97cf5035b 优化 2025-06-20 14:56:17 +08:00
沈健 9d1a9a769e 优化 2025-06-20 14:53:54 +08:00
沈健 399151ff58 优化企业开发区同步 2025-06-20 14:48:16 +08:00
沈健 9ce31e9b56 优化企业开发区同步 2025-06-20 13:30:38 +08:00
沈健 3975d029f8 优化企业开发区同步 2025-06-20 13:28:52 +08:00
沈健 d0e6714ecd 优化企业开发区同步 2025-06-20 13:18:31 +08:00
沈健 d0ebcb451a 优化企业开发区同步 2025-06-20 13:16:24 +08:00
沈健 663157757b 优化企业开发区同步 2025-06-20 13:14:21 +08:00
沈健 405f40f0b8 Merge remote-tracking branch 'origin/master' 2025-06-20 13:10:01 +08:00
沈健 1605ef8165 优化 2025-06-20 13:09:37 +08:00
admin 5cd3470091 Merge remote-tracking branch 'origin/master' 2025-06-19 16:43:40 +08:00
admin 7363524890 恢复加工数据到原始数据 2025-06-19 16:41:40 +08:00
沈健 241de34a92 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2025-06-19 16:16:40 +08:00
沈健 51fda5fdf2 优化 2025-06-19 16:16:32 +08:00
admin ffafaa919c 优化楼盘评级-优化排名算法 2025-06-19 15:27:41 +08:00
admin 6c69efaed4 优化楼盘评级-优化排名算法 2025-06-19 14:52:21 +08:00
admin e150527534 优化楼盘评级-优化排名算法 2025-06-19 13:37:01 +08:00
沈健 4868577bef Merge remote-tracking branch 'origin/master' 2025-06-19 09:36:13 +08:00
沈健 b5583f4e66 优化 2025-06-19 09:35:45 +08:00
admin d3b666d3e7 Merge remote-tracking branch 'origin/master' 2025-06-18 18:28:45 +08:00
admin 4572a84568 优化楼盘评级-优化排名算法 2025-06-18 18:28:25 +08:00
沈健 a16b7e4404 Merge remote-tracking branch 'origin/master' 2025-06-18 14:06:00 +08:00
沈健 3a00ac99c1 优化 2025-06-18 14:05:26 +08:00
admin 5562054ea3 Merge remote-tracking branch 'origin/master' 2025-06-18 11:52:34 +08:00
admin b70761874a 优化楼盘评级-优化排名算法 2025-06-18 11:52:10 +08:00
沈健 b3fbf74e5c 暂时注释 2025-06-18 11:23:55 +08:00
沈健 8b9ff84b64 优化 2025-06-18 11:20:40 +08:00
沈健 e01efaca40 优化 2025-06-18 10:58:27 +08:00
沈健 f08fadb7c9 Merge remote-tracking branch 'origin/master' 2025-06-18 10:50:25 +08:00
沈健 e079a20472 优化 2025-06-18 10:50:02 +08:00
admin 27319ed28d 优化楼盘评级-多进程处理 2025-06-18 10:36:43 +08:00
admin cea2711e5f Merge remote-tracking branch 'origin/master' 2025-06-18 09:52:57 +08:00
admin f36d7bd51f 优化楼盘评级 2025-06-18 09:52:34 +08:00
沈健 7b0542a36a Merge remote-tracking branch 'origin/master' 2025-06-18 09:44:03 +08:00
沈健 b1a7892268 修复无围栏数据 2025-06-18 09:43:12 +08:00
admin 235fd10e3f 优化楼盘评级 2025-06-17 17:15:51 +08:00
admin 1f04a0a198 优化楼盘评级 2025-06-17 17:07:55 +08:00
admin ddaccd7600 优化楼盘评级 2025-06-17 16:34:03 +08:00
admin 36b4cc59a9 优化楼盘评级 2025-06-16 18:38:58 +08:00
admin 22c2711485 优化楼盘评级 2025-06-16 18:32:44 +08:00
admin 3a7c6b683c 优化楼盘评级 2025-06-16 18:11:52 +08:00
admin 019d59ef51 优化楼盘评级 2025-06-16 18:08:18 +08:00
admin 9750eb7697 Merge remote-tracking branch 'origin/master' 2025-06-16 18:02:05 +08:00
admin 48ad7629a3 优化楼盘评级 2025-06-16 18:01:37 +08:00
沈健 b6502d0051 修复重算BUG 2025-06-16 13:32:02 +08:00
沈健 e60ec4dfaa Merge remote-tracking branch 'origin/master' 2025-06-12 13:26:48 +08:00
沈健 bc7f275d06 修复重算BUG 2025-06-12 13:26:27 +08:00
沈健 a6e60819ad 修复重算BUG 2025-06-12 13:17:16 +08:00
沈健 92ce23a11d 新增写字楼出售价格重算 2025-06-10 16:38:08 +08:00
沈健 6ce61b6336 新增写字楼出售价格重算 2025-06-10 16:33:31 +08:00
沈健 96c795be4a 新增写字楼出售价格重算 2025-06-10 16:27:26 +08:00
沈健 df71f89b07 新增写字楼出售价格重算 2025-06-10 15:31:57 +08:00
沈健 cb938f81f0 还原 2025-06-10 15:12:44 +08:00
沈健 77cc4fb096 优化 2025-06-10 14:46:52 +08:00
沈健 24cbd81454 优化 2025-06-10 14:41:41 +08:00
沈健 43e9b5af14 优化 2025-06-10 14:04:11 +08:00
沈健 a09c8b2feb 优化 2025-06-10 09:06:10 +08:00
沈健 ac357b1d9e 优化 2025-06-09 14:52:19 +08:00
沈健 0d8c221dd3 优化 2025-06-09 14:49:24 +08:00
沈健 6375437b7f 优化 2025-06-09 14:46:03 +08:00
沈健 0fee4586b2 Merge remote-tracking branch 'origin/master' 2025-06-09 14:36:46 +08:00
沈健 351bad8b57 优化 2025-06-09 14:36:20 +08:00
admin 04ab71fc1c Merge remote-tracking branch 'origin/master' 2025-06-09 13:26:47 +08:00
admin 6ba24850fc 修复价格数据为空bug 2025-06-09 13:26:27 +08:00
沈健 b4b1a39118 优化 2025-06-09 11:36:55 +08:00
沈健 2f3a64a018 优化 2025-06-09 11:35:51 +08:00
沈健 dc67eeba57 优化 2025-06-09 11:35:01 +08:00
沈健 2428254918 Merge remote-tracking branch 'origin/master' 2025-06-09 11:33:24 +08:00
沈健 7409e0edcf 优化 2025-06-09 11:32:59 +08:00
admin 2ddec57ba3 修改city参数 2025-06-09 11:23:12 +08:00
admin 982f42fe14 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2025-06-09 11:16:07 +08:00
admin 338870334c 修改楼盘评级时间 2025-06-09 11:15:54 +08:00
沈健 7072183435 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2025-06-09 10:50:55 +08:00
沈健 43f35fa63e 新增企业库处理流程 2025-06-09 10:50:30 +08:00
admin b064138c46 添加楼盘评级算法 2025-06-09 10:48:21 +08:00
沈健 02e1aab542 新增百度转高德 2025-06-06 15:23:05 +08:00
沈健 5e6ae745a1 优化写入禁止立即刷新 2025-06-06 10:54:02 +08:00
沈健 ca1cf0f15c 优化写入禁止立即刷新 2025-06-06 10:52:17 +08:00
沈健 20b606a0dd 优化 2025-06-05 17:47:27 +08:00
沈健 27de5a7ec9 优化 2025-06-05 17:36:32 +08:00
沈健 e3e3ebc981 优化 2025-06-05 17:31:07 +08:00
沈健 f502d7f05e 优化 2025-06-05 17:28:57 +08:00
沈健 49a387f9ed 优化 2025-06-05 17:22:32 +08:00
沈健 4bb93c977f 优化 2025-06-05 17:19:49 +08:00
沈健 7d9d16d33a 优化 2025-06-05 17:09:56 +08:00
沈健 237e88d79f 优化 2025-06-05 17:08:03 +08:00
沈健 165e50ae16 修復sys报错 2025-06-05 17:02:48 +08:00
沈健 c4368c1421 修復sys报错 2025-06-05 16:59:52 +08:00
majintao ba1d065337 财力评级改成城市简称 2025-06-03 13:47:36 +08:00
沈健 f30f940e0b Merge remote-tracking branch 'origin/master' 2025-05-27 10:55:28 +08:00
沈健 b9d9e728ad 修復sys报错 2025-05-27 10:54:21 +08:00
majintao c8acedbd14 财力评级从mongo查学校、医院 2025-05-22 18:11:00 +08:00
沈健 82119764c6 优化 2025-04-27 10:39:26 +08:00
沈健 82f7359410 改为周一到周五运行 2025-04-27 09:23:53 +08:00
沈健 16aea01faf 修复商铺均价BUG 2025-04-25 16:36:03 +08:00
沈健 72106ba0ae 优化 2025-04-24 09:28:56 +08:00
沈健 2728bbbdad 优化 2025-04-23 17:54:22 +08:00
沈健 bc7a02d319 优化 2025-04-23 17:50:31 +08:00
沈健 47bb142987 优化端口 2025-04-23 16:52:35 +08:00
沈健 d6df49c9a6 优化 2025-04-23 16:09:10 +08:00
沈健 4b01ed8fd6 优化 2025-04-23 16:06:58 +08:00
沈健 c4566e8426 优化 2025-04-23 16:04:52 +08:00
沈健 cde3fd5e40 优化 2025-04-23 16:04:03 +08:00
沈健 9962cf5ef3 优化 2025-04-23 15:47:44 +08:00
沈健 e5ceaddfc4 优化 2025-04-23 15:44:45 +08:00
沈健 f8c27a33a2 优化 2025-04-23 15:41:41 +08:00
沈健 66bb733a0b 优化 2025-04-23 15:36:19 +08:00
沈健 d49b907f38 优化 2025-04-23 15:34:02 +08:00
沈健 7c8a704922 优化 2025-04-23 15:30:39 +08:00
沈健 46a7501cca TEST 2025-04-23 15:23:00 +08:00
沈健 9881b93d7f TEST 2025-04-23 15:20:23 +08:00
沈健 5b67698197 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow
# Conflicts:
#	dags/jh/price/price_deviation.py
2025-04-23 14:53:55 +08:00
沈健 db93c0fef2 修改库 2025-04-23 14:34:54 +08:00
沈健 f1a8e3e001 修改库 2025-04-23 14:27:26 +08:00
沈健 7f91150550 新增AI 2025-04-23 13:27:09 +08:00
沈健 f4b49bd870 Merge remote-tracking branch 'origin/master'
# Conflicts:
#	dags/ai/stock/stock.py
2025-04-23 13:25:36 +08:00
沈健 744ef7c9f5 新增AI 2025-04-23 13:25:08 +08:00
沈健 ef8161abe8 新增AI 2025-04-23 13:23:57 +08:00
沈健 fb33ddc969 新增告警 2025-04-22 14:38:43 +08:00
沈健 80b4867cf3 新增告警 2025-04-22 14:34:54 +08:00
沈健 3e72a9a6af 优化范围统计 2025-04-17 16:34:45 +08:00
沈健 6fe9688eac 优化范围统计 2025-04-17 16:31:45 +08:00
沈健 bc43c1457f 等级改为6 2025-04-17 15:36:50 +08:00
沈健 916804115b 新增多城市支持 2025-04-17 09:50:40 +08:00
沈健 0e2f9d0cae Merge remote-tracking branch 'origin/master'
# Conflicts:
#	dags/business/business/community/consumption.py
2025-04-16 19:03:58 +08:00
沈健 fb8f307596 优化分数 2025-04-16 19:03:45 +08:00
沈健 3c88276173 优化分数 2025-04-15 18:23:32 +08:00
沈健 713e6ce9ba 注释临时 2025-04-15 09:57:33 +08:00
沈健 9968c6ed0c Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2025-04-15 09:53:40 +08:00
沈健 390b869d3e 优化执行 2025-04-15 09:52:24 +08:00
沈健 88b1e939ff 优化 2025-04-14 17:57:04 +08:00
沈健 80d391b358 命名修改 2025-04-14 14:35:02 +08:00
沈健 acc10c6c3d 修复BUG 2025-04-14 14:16:45 +08:00
沈健 c45f480a42 优化报错警告 2025-04-14 14:02:30 +08:00
沈健 3c77ac061c 新增子因子评分 2025-04-14 11:44:33 +08:00
沈健 027ba5ef1f 新增子因子评分 2025-04-14 11:14:13 +08:00
沈健 6b4e6ee71e 优化财力评级 2025-04-14 10:56:06 +08:00
沈健 146fe2e680 优化财力评级 2025-04-11 18:52:48 +08:00
沈健 ddfb1cde70 优化财力评级 2025-04-11 15:06:50 +08:00
沈健 608f492b29 优化少量城市评级 2025-04-11 12:18:44 +08:00
沈健 53e7ba8c99 优化少量城市评级 2025-04-11 12:14:40 +08:00
沈健 4a2e34c25f 优化少量城市评级 2025-04-11 12:13:04 +08:00
沈健 6c6841988e 优化少量城市评级 2025-04-11 10:02:20 +08:00
沈健 82c3191983 优化财力评级补充 2025-04-10 18:20:40 +08:00
沈健 88e6099e0b 优化 2025-04-10 17:18:18 +08:00
沈健 83dfdb7ab0 Merge remote-tracking branch 'origin/master' 2025-04-10 16:16:43 +08:00
liuhongyang 9d2c12653d 财力评级统计优化 2025-04-10 11:34:02 +08:00
沈健 7d25815e7e 新增商铺出售 2025-04-10 11:10:38 +08:00
沈健 747c79eef2 优化 2025-04-10 10:08:20 +08:00
沈健 51e4abdcfc Merge remote-tracking branch 'origin/master' 2025-04-10 09:43:04 +08:00
沈健 b61199d0be 新增商铺价格计算 2025-04-10 09:41:46 +08:00
liuhongyang 153a1aa418 财力评级统计优化 2025-04-09 16:07:42 +08:00
liuhongyang f943abb51e 财力评级统计优化 2025-04-09 15:47:46 +08:00
沈健 12f1dcf7cf 优化统计 2025-04-08 18:15:44 +08:00
沈健 f06263e434 优化全量统计 2025-04-08 18:05:42 +08:00
沈健 c1e55c08a4 优化全量统计 2025-04-08 17:58:48 +08:00
沈健 d183ca5c6c 优化全量统计 2025-04-08 17:51:45 +08:00
liuhongyang c936b2c2c5 财力评级统计 2025-04-08 17:45:36 +08:00
沈健 8723806fab you 2025-04-08 15:35:40 +08:00
沈健 56519e159c 新增均价计算类型 2025-04-08 14:21:02 +08:00
沈健 c640d914cc 物业费获取方式 2025-04-08 13:55:22 +08:00
沈健 b0708d3a3e 新增统计 2025-04-07 17:02:37 +08:00
沈健 319ef5847c 优化报错 2025-04-07 09:45:38 +08:00
沈健 43118a7267 新增财力评级统计 2025-04-03 15:22:16 +08:00
沈健 9f8a34ac8d 新增评级总分100 2025-04-03 14:22:36 +08:00
沈健 db3c29c3c9 新增评级总分100 2025-04-03 14:17:48 +08:00
沈健 7b17928545 新增评级总分100 2025-04-03 13:56:56 +08:00
沈健 ca4b7782ef 评级总分临时改为100(支付宝) 2025-04-02 14:04:40 +08:00
沈健 dcce114957 优化价格类型 2025-03-31 15:56:28 +08:00
沈健 515351d56b Merge remote-tracking branch 'origin/master'
# Conflicts:
#	dags/office/sell/fill_price.py
2025-03-26 15:18:04 +08:00
沈健 8eb6cb3876 新增城市区域均价补充 2025-03-26 15:16:07 +08:00
沈健 b36485473c 新增城市区域均价补充 2025-03-26 15:00:28 +08:00
沈健 812888eb79 新增写字楼售价填充 2025-03-26 13:31:39 +08:00
沈健 0c751f413a 新增写字楼售价填充 2025-03-26 13:25:57 +08:00
沈健 46515fe238 新增城市区域均价计算 2025-03-25 09:49:10 +08:00
沈健 a91fe73747 新增出售均价算法 2025-03-24 17:47:29 +08:00
沈健 61f39b36b6 新增出售均价算法 2025-03-24 17:43:49 +08:00
沈健 7f38681f70 新增出售均价算法 2025-03-24 17:13:44 +08:00
沈健 13dabf5682 优化 2025-03-21 14:47:03 +08:00
沈健 81438e9d39 优化 2025-03-21 14:33:14 +08:00
沈健 8a3b2fdcf4 优化 2025-03-21 09:13:53 +08:00
沈健 0067fc6775 历史数据跑批 2025-03-21 09:09:57 +08:00
沈健 4f9fe0dce9 租金BUG修复 2025-03-20 15:20:48 +08:00
沈健 7568cefde6 租金BUG修复 2025-03-20 14:45:09 +08:00
沈健 15c3ec9857 新增写字楼出售计算 2025-03-13 17:13:20 +08:00
沈健 401cc4fbe9 新增写字楼出售计算 2025-03-13 13:02:41 +08:00
沈健 bb217cc4c3 新增m_date周边 2025-03-11 17:46:19 +08:00
沈健 0fa69ae9c0 优化 2025-03-07 17:35:52 +08:00
沈健 5f9c42623c 优化计算方式 2025-03-07 15:55:10 +08:00
沈健 079652a557 修改为5个 2025-03-07 15:14:57 +08:00
沈健 7df202a638 优化报错 2025-03-06 13:05:45 +08:00
沈健 6fe4240883 优化报错 2025-03-06 11:35:48 +08:00
沈健 fd586603d9 优化报错 2025-03-06 11:19:01 +08:00
沈健 2e806a81cb 支持城市传参 2025-03-06 10:39:41 +08:00
沈健 50e30d9dc0 修复BUG 2025-03-06 10:25:07 +08:00
沈健 0c6a7ad583 修复户均总价噪音影响 2025-03-05 14:56:02 +08:00
沈健 e3579708e4 新增子评级 2025-03-05 11:01:49 +08:00
沈健 d89ecf29f0 优化定时 2025-03-03 09:46:32 +08:00
沈健 5b5252c185 优化定时 2025-03-03 09:44:23 +08:00
沈健 3a30ed8880 优化更新 2025-02-28 17:22:24 +08:00
沈健 ad778373e4 优化更新 2025-02-28 17:19:28 +08:00
沈健 c0e2f6aaad 优化更新 2025-02-28 16:41:54 +08:00
沈健 92f7ca4362 优化更新 2025-02-28 16:17:18 +08:00
沈健 95dea214e7 新增写字楼计算 2025-02-28 15:58:37 +08:00
沈健 a091466906 新增写字楼系数计算逻辑 2025-02-26 10:27:16 +08:00
沈健 241597d1ae 优化 2025-02-17 14:55:00 +08:00
沈健 c2cf59acb8 Merge remote-tracking branch 'origin/master' 2025-02-13 13:18:23 +08:00
沈健 2d6ff5e11c 新增农行定时 2025-02-13 13:18:15 +08:00
沈健 2d68181688 新增农行定时 2025-02-13 13:13:35 +08:00
沈健 a78c8010fc 新增农行定时 2025-02-13 13:10:16 +08:00
沈健 0202976699 优化 2025-02-12 12:03:11 +08:00
沈健 bcca0c0311 修复区域价格NAN问题 2025-01-22 16:34:18 +08:00
沈健 8bf59f7b50 修复价格NAN问题 2025-01-22 14:46:13 +08:00
沈健 ecd912f8e6 优化 2025-01-21 15:37:59 +08:00
沈健 12bcea225a 优化 2025-01-21 15:36:49 +08:00
沈健 b6f79e808f 优化 2025-01-21 15:34:31 +08:00
沈健 72fcacd73f 优化 2025-01-21 15:30:46 +08:00
沈健 a226629f9c 新增写字楼均价计算 2025-01-21 15:29:15 +08:00
沈健 cb469f2f41 写字楼租金模型beta 2025-01-17 18:40:44 +08:00
沈健 528a257113 周边配套计算库修改 2024-12-23 15:26:44 +08:00
gallup 075577afd7 提示添加 2024-12-06 10:49:09 +08:00
gallup b8b2680b00 修改说明 2024-12-06 10:16:40 +08:00
gallup 5913f8fff0 区域房价指数和小区房价指数定时计算 2024-12-06 10:12:29 +08:00
sjshenjian db8cbf0edb 修复BUG 2024-11-27 16:33:40 +08:00
sjshenjian 1cd5079ac4 优化 2024-11-21 10:48:38 +08:00
sjshenjian 67eed5a9df 周边POI改为多线程 2024-11-20 19:00:55 +08:00
sjshenjian 3ed3bf7641 调整顺序 2024-11-14 17:50:23 +08:00
sjshenjian b67bd8bd1e 改造周边配套 2024-11-14 16:14:16 +08:00
sjshenjian 7ab06bd585 优化 2024-11-11 14:23:36 +08:00
sjshenjian 9172e1a4f9 修复建筑面积问题 2024-11-11 11:22:30 +08:00
sjshenjian b07f22cf33 优化安装goojia_utils效率 2024-10-24 10:58:29 +08:00
sjshenjian 00e258e4a8 优化 2024-10-24 10:57:18 +08:00
沈健 da778f9a02 新增小区规模计算 2024-10-11 10:44:13 +08:00
沈健 a82c958915 新增小区规模计算 2024-09-25 14:53:02 +08:00
majintao 850ba31f49 API账号到期短信发送 2024-09-24 13:18:17 +08:00
majintao a97f0262b1 API账号到期短信发送 2024-09-24 13:12:04 +08:00
majintao d6beb9df39 API用户使用及明细的邮件发送 2024-09-23 17:33:29 +08:00
admin 937d370b0b Merge remote-tracking branch 'origin/master' 2024-09-10 16:55:27 +08:00
admin 1bbf0c3fb6 优化成交挂牌价系数 2024-09-10 16:55:18 +08:00
沈健 8f6744966d 优化 2024-08-26 11:45:15 +08:00
沈健 8d07f4daff 优化 2024-08-26 10:15:16 +08:00
沈健 aa0f3719a5 财力评级206改为89 2024-08-26 10:10:11 +08:00
沈健 ac8efee40b 财力评级改为89 2024-08-26 10:04:03 +08:00
沈健 2d7aaf647f 新增其他字段百分位计算 2024-08-26 09:55:06 +08:00
admin 9b962728e6 优化成交挂牌价系数 2024-08-16 12:59:57 +08:00
admin 3408b712cf 优化成交挂牌价系数 2024-08-16 12:58:01 +08:00
admin 9655f2cbe5 优化成交挂牌价系数 2024-08-16 12:53:27 +08:00
admin a9abba222d 优化成交挂牌价系数 2024-08-16 11:32:24 +08:00
admin aebb2bd467 优化成交挂牌价系数 2024-08-16 11:26:34 +08:00
admin 9f82d9e0f4 优化成交挂牌价系数 2024-08-16 11:06:29 +08:00
admin 4c55387790 优化成交挂牌价系数 2024-08-16 10:33:54 +08:00
admin 6fe480ee66 优化成交挂牌价系数 2024-08-15 17:46:01 +08:00
admin f113be7ba0 Merge remote-tracking branch 'origin/master' 2024-08-15 17:39:40 +08:00
admin 1194467e5f 优化成交挂牌价系数 2024-08-15 17:39:33 +08:00
沈健 6e56ac4c02 优化 2024-08-14 18:13:19 +08:00
沈健 1f60c344bd 优化 2024-08-14 18:09:40 +08:00
沈健 7e60e06c3f 优化 2024-08-14 18:08:10 +08:00
沈健 eb0e37884b Merge remote-tracking branch 'origin/master' 2024-08-14 16:26:50 +08:00
沈健 cda9cd919a 新增OPEN日志监控 2024-08-14 16:26:32 +08:00
admin 7dc6a7eea6 添加楼盘评级算法 2024-08-14 09:21:05 +08:00
admin 3289317575 Merge remote-tracking branch 'origin/master' 2024-08-14 09:20:18 +08:00
admin cc2a198a83 添加楼盘评级算法 2024-08-14 09:20:04 +08:00
沈健 a4fc57a52a 新增删除逻辑 2024-08-12 10:21:46 +08:00
沈健 3bb8da4722 新增删除逻辑 2024-08-12 10:19:13 +08:00
沈健 10f2e97cd2 还原 2024-08-02 18:48:08 +08:00
沈健 1de9f959e4 暂时写死黑龙江省 2024-08-02 16:20:26 +08:00
沈健 3ffb594338 暂时写死 2024-08-02 15:54:29 +08:00
沈健 5c3d38f171 Merge remote-tracking branch 'origin/master' 2024-08-02 15:29:40 +08:00
沈健 00c27ba1c1 优化 2024-08-02 15:29:29 +08:00
muliangshu 446088c7d9 第三方均价输出 2024-07-23 10:58:51 +08:00
muliangshu 9f8aa6c7a7 第三方均价输出 2024-07-23 10:57:45 +08:00
muliangshu 0bdbed9454 第三方均价输出 2024-07-23 10:50:10 +08:00
沈健 0f44db29cb 优化 2024-07-22 17:48:36 +08:00
沈健 0fb7e9186f 优化 2024-07-22 17:45:21 +08:00
沈健 32378a2073 优化 2024-07-22 17:41:36 +08:00
沈健 ad255ba0db 优化 2024-07-22 17:38:13 +08:00
沈健 202a5b9198 优化 2024-07-22 17:37:38 +08:00
muliangshu 89c472d131 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2024-07-22 17:13:59 +08:00
muliangshu 3cda5cd211 第三方均价输出 2024-07-22 17:13:48 +08:00
admin 5d94eb7bd0 添加楼盘评级算法 2024-07-15 10:35:53 +08:00
admin 5dd1991a18 添加楼盘评级算法 2024-07-15 09:38:20 +08:00
admin 9f9c3b6555 添加楼盘评级算法 2024-07-12 16:38:33 +08:00
admin 058ed8448d 添加楼盘评级算法 2024-07-12 16:21:46 +08:00
admin 6c311a3e05 添加楼盘评级算法 2024-07-12 16:04:17 +08:00
sjshenjian aafafccb28 新增省市区全拼转换 2024-07-09 16:28:43 +08:00
sjshenjian 840eb8408a 新增省市区全拼转换 2024-07-09 16:11:34 +08:00
sjshenjian 16999f22b2 新增省市区全拼转换 2024-07-09 16:07:55 +08:00
sjshenjian 8c5b16e519 总评级修改 2024-07-09 15:39:34 +08:00
沈健 5fdb0ac7ce Merge remote-tracking branch 'origin/master' 2024-05-31 15:53:26 +08:00
沈健 7c8dbeed6c 优化 2024-05-31 15:53:17 +08:00
mjt 1c24b23e63 城市区域价格涨跌幅 2024-05-30 17:25:55 +08:00
mjt 5ac62440b1 城市区域价格涨跌幅 2024-05-30 17:23:09 +08:00
mjt 52716c3778 Merge remote-tracking branch 'origin/master' 2024-05-30 17:19:26 +08:00
mjt 668eb29a8d 城市区域价格涨跌幅 2024-05-30 17:19:13 +08:00
沈健 c360db45c9 屏蔽过高物业费问题 2024-05-29 17:58:41 +08:00
沈健 728dfa0e94 屏蔽过高物业费问题 2024-05-29 17:00:02 +08:00
沈健 4cc53dfbf5 新增独家小区输出 2024-05-20 13:16:59 +08:00
沈健 9601ede115 新增差异值 2024-05-17 16:48:00 +08:00
gaojuan 24d7200b75 检查文件名修改 2024-05-09 16:06:51 +08:00
gaojuan 5d8397ec85 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2024-03-08 21:37:59 +08:00
gaojuan 8c5779fd07 update 2024-03-08 21:37:47 +08:00
gaojuan 2fb76d1f55 加2个文件验证 2024-03-08 21:37:05 +08:00
sunqiang 726490f872 测试偏差较大数据展示PPE达标的成交案例 2023-11-30 10:43:18 +08:00
sunqiang fdedc4cbf2 pd.read换成pgsql.scan_by_sql_to_data_frame 2023-11-26 12:00:19 +08:00
sunqiang b761162d28 去除con=engine 2023-11-25 20:18:32 +08:00
sunqiang cb2ab3c418 测试偏差较大案例统计数据新增 房价网均价、链家均价、贝壳均价 2023-11-25 17:58:56 +08:00
gaojuan 3146952bd1 文件名修改 2023-10-31 14:45:04 +08:00
zhaojixiang@fangjia.com 6ac82bce70 修改错误 2023-08-02 14:36:00 +08:00
zhaojixiang@fangjia.com a6c7076e14 修复错误 2023-08-02 13:21:22 +08:00
zhaojixiang@fangjia.com 780f4e469d 修改房源网站统计规则,改为前70个热点城市当月数量少于100条提醒 2023-08-02 09:40:38 +08:00
zhaojixiang@fangjia.com f03bc87f6f 修改区域维度统计周期,改为2个月 2023-08-01 15:20:20 +08:00
zhaojixiang@fangjia.com 339c84ea24 设置执行时间,每月10号0点 2023-08-01 13:14:45 +08:00
zhaojixiang@fangjia.com c4e9f38b51 新增房源样本监控脚本 2023-08-01 10:20:07 +08:00
zhaojixiang@fangjia.com 18ee0b05ac 新增API接口监控提醒任务 2023-07-10 17:31:15 +08:00
zhaojixiang@fangjia.com 2723cb6ef0 新增API接口监控提醒任务 2023-07-10 17:06:30 +08:00
zhaojixiang@fangjia.com 8fd13e2dd8 Merge remote-tracking branch 'origin/master' 2023-07-05 09:51:28 +08:00
zhaojixiang@fangjia.com 278206fab5 db 2023-07-05 09:39:49 +08:00
gaojuan 35cb0fdbcb Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2023-06-09 17:27:13 +08:00
gaojuan 2d8465c219 融合10号第三方不检测 2023-06-09 17:27:05 +08:00
Jian Shen 83665e7e81 优化 2023-06-08 15:03:10 +08:00
shenjian@fangjia.com 6fee8096dd db 2023-06-01 16:44:58 +08:00
shenjian@fangjia.com 7f8e044640 db 2023-06-01 16:25:15 +08:00
shenjian@fangjia.com 5192fbd93c db 2023-06-01 15:28:56 +08:00
shenjian@fangjia.com 89e5814468 优化镜像拉取 2023-05-29 15:28:47 +08:00
shenjian@fangjia.com 6b4cb7f78a 优化镜像拉取 2023-05-29 15:26:20 +08:00
shenjian@fangjia.com 5877566c2e 优化 2023-05-22 16:07:34 +08:00
shenjian@fangjia.com 2f267ee038 注释 2023-05-16 16:50:57 +08:00
shenjian@fangjia.com d82c2f1b2b 回退相对价格排名优化 2023-05-16 16:20:00 +08:00
shenjian@fangjia.com 036a6791a7 新增城市等级级价格等级评级 2023-05-15 17:30:08 +08:00
shenjian@fangjia.com fab0bb6d6d Merge remote-tracking branch 'origin/master'
# Conflicts:
#	dags/business/business/community/city_level_and_price_rating.py
2023-05-15 15:47:15 +08:00
shenjian@fangjia.com 25f30657bf 新增城市等级级价格等级评级 2023-05-15 15:46:17 +08:00
shenjian@fangjia.com 100d89e556 新增城市等级级价格等级评级 2023-05-15 15:33:02 +08:00
shenjian@fangjia.com 95462de314 新增城市等级级价格等级评级 2023-05-15 15:16:16 +08:00
shenjian@fangjia.com 0077622e63 每月20日自动话入库 2023-05-11 10:11:21 +08:00
shenjian@fangjia.com 58ec14eb1b 优化 2023-05-11 09:57:41 +08:00
shenjian@fangjia.com 33f65f6199 优化 2023-05-11 09:48:27 +08:00
shenjian@fangjia.com 3801ddcc70 优化 2023-05-11 09:36:28 +08:00
shenjian@fangjia.com 787380ad94 优化 2023-05-11 09:32:05 +08:00
shenjian@fangjia.com 4e0deb2159 优化 2023-05-10 17:03:25 +08:00
shenjian@fangjia.com 8760b4d513 优化 2023-05-10 16:54:14 +08:00
shenjian@fangjia.com 6cfd477e30 新增格子 2023-05-10 16:46:19 +08:00
shenjian@fangjia.com f6faf38ab0 新增格子 2023-05-10 16:38:49 +08:00
shenjian@fangjia.com 27c5fcacc1 新增格子 2023-05-10 16:09:21 +08:00
shenjian@fangjia.com 6d35355baf 修改财力评级计算方式 2023-05-10 15:24:48 +08:00
shenjian@fangjia.com 1a921c08fd 优化 2023-05-10 15:18:40 +08:00
shenjian@fangjia.com d4a0ecc971 优化 2023-05-10 15:16:48 +08:00
shenjian@fangjia.com bc1af1c4db 优化 2023-05-10 15:13:52 +08:00
shenjian@fangjia.com 0fa6b8821f 优化 2023-05-10 15:13:06 +08:00
shenjian@fangjia.com db23742520 优化 2023-05-10 11:20:55 +08:00
shenjian@fangjia.com ac063da374 优化 2023-05-10 11:16:58 +08:00
shenjian@fangjia.com c9ee9a2381 优化 2023-05-10 11:16:13 +08:00
shenjian@fangjia.com 7adb2a18a7 优化 2023-05-10 11:05:35 +08:00
shenjian@fangjia.com 117affd8ce 优化 2023-05-10 11:03:48 +08:00
shenjian@fangjia.com 48de7df4b2 优化 2023-05-10 10:52:49 +08:00
shenjian@fangjia.com 98c1ad5baa 传参 2023-05-10 09:45:48 +08:00
shenjian@fangjia.com 6c3ef4ebbe 传参 2023-05-10 09:40:42 +08:00
shenjian@fangjia.com 7313a03f76 传参 2023-05-10 09:34:16 +08:00
Jian Shen a4861702c6 优化 2023-05-09 21:39:28 +08:00
Jian Shen 4f16894602 优化 2023-05-09 21:36:35 +08:00
Jian Shen 19ff0909f5 优化 2023-05-09 21:28:13 +08:00
Jian Shen 3c2884bee9 优化 2023-05-09 21:22:42 +08:00
shenjian@fangjia.com 89ef0a32e2 传参 2023-05-09 17:45:51 +08:00
shenjian@fangjia.com 4f6853f710 传参 2023-05-09 17:44:45 +08:00
shenjian@fangjia.com f8c67d5e9f 传参 2023-05-09 17:41:56 +08:00
shenjian@fangjia.com 2ed24c67b7 传参 2023-05-09 17:30:46 +08:00
shenjian@fangjia.com 8174eaa78d docker 2023-05-09 16:58:20 +08:00
shenjian@fangjia.com 35c374fc46 docker 2023-05-09 16:56:10 +08:00
shenjian@fangjia.com 09444c04b7 放开 2023-05-09 16:52:16 +08:00
shenjian@fangjia.com cf6508a2c5 优化城市无均价报错 2023-05-09 16:45:50 +08:00
shenjian@fangjia.com a9392e1337 优化城市无均价报错 2023-05-09 16:40:16 +08:00
shenjian@fangjia.com d2fc14b43a 优化检测 2023-05-09 16:25:23 +08:00
shenjian@fangjia.com f98a75cc2d 新增格子均价 2023-05-09 14:11:48 +08:00
shenjian@fangjia.com 43038a0478 新增格子均价 2023-05-09 13:50:54 +08:00
shenjian@fangjia.com 4a44be7c29 新增格子均价 2023-05-09 13:49:53 +08:00
shenjian@fangjia.com 279663f351 新增格子均价 2023-05-09 13:43:06 +08:00
shenjian@fangjia.com 3da15c2b5c 新增格子均价 2023-05-09 13:18:04 +08:00
shenjian@fangjia.com 13bb857701 新增格子均价 2023-05-09 13:17:24 +08:00
shenjian@fangjia.com 83cdba3406 新增格子均价 2023-05-09 12:10:25 +08:00
shenjian@fangjia.com 40aad80afe 新增格子均价 2023-05-09 11:05:50 +08:00
shenjian@fangjia.com b3910631db 优化 2023-05-08 18:24:28 +08:00
shenjian@fangjia.com e1b9abbda3 优化 2023-05-08 18:23:32 +08:00
shenjian@fangjia.com d4dfcb7146 优化 2023-05-08 18:22:39 +08:00
shenjian@fangjia.com 0611777787 优化 2023-05-08 17:40:08 +08:00
admin 456fd66319 新增自动化入库定时 2023-05-08 17:21:06 +08:00
shenjian@fangjia.com 66dec65a06 优化 2023-05-06 12:23:05 +08:00
Jian Shen 8598b007ae 优化 2023-04-27 21:05:41 +08:00
shenjian@fangjia.com b061834407 优化 2023-04-27 18:26:19 +08:00
shenjian@fangjia.com d3386f5b10 优化 2023-04-27 13:11:23 +08:00
shenjian@fangjia.com 499ee47be0 优化 2023-04-27 12:27:03 +08:00
shenjian@fangjia.com ee60711af1 优化 2023-04-27 11:34:52 +08:00
shenjian@fangjia.com ee82ee0846 优化 2023-04-27 11:33:46 +08:00
Jian Shen 0792dccd92 优化 2023-04-25 22:17:40 +08:00
Jian Shen 13747baa12 优化 2023-04-25 22:15:27 +08:00
Jian Shen 6a0d4149b8 优化 2023-04-25 21:48:36 +08:00
Jian Shen 2d0c87b701 优化 2023-04-25 20:05:21 +08:00
shenjian@fangjia.com 4818ce566a 新增同步字段 2023-04-25 18:09:40 +08:00
shenjian@fangjia.com 8ab375aa5d 新增同步字段 2023-04-25 18:00:57 +08:00
shenjian@fangjia.com f4f5c86e1e 新增同步字段 2023-04-25 16:15:59 +08:00
shenjian@fangjia.com bdc9a3c62a 新增同步字段 2023-04-25 14:43:14 +08:00
shenjian@fangjia.com db2b2005dd 新增同步字段 2023-04-25 13:54:28 +08:00
shenjian@fangjia.com 29f827e2cc 解决airflow苏州城市price同步bug 2023-04-25 13:24:21 +08:00
shenjian@fangjia.com 2f7478f017 新增错误重试 2023-04-24 18:23:40 +08:00
shenjian@fangjia.com 9c07c89801 bak 2023-04-23 14:07:47 +08:00
shenjian@fangjia.com 767a0d040e bak 2023-04-23 13:55:27 +08:00
shenjian@fangjia.com 0b5dff7111 bak 2023-04-23 13:51:39 +08:00
shenjian@fangjia.com b915e1fe1b bak 2023-04-21 11:35:25 +08:00
shenjian@fangjia.com 2a9fa3574f CK导库 2023-04-20 18:39:21 +08:00
shenjian@fangjia.com 6ef0d4adfc Merge remote-tracking branch 'origin/master' 2023-04-20 16:29:03 +08:00
shenjian@fangjia.com c3d937e326 新增覆盖率 2023-04-20 16:28:07 +08:00
shenjian@fangjia.com 74e3211fb7 建表备份 2023-04-20 13:31:08 +08:00
Jian Shen 0b912ab106 新增户均总价中位值 2023-04-15 12:13:13 +08:00
shenjian@fangjia.com d1e5b5579d 修复美团csv问题 2023-04-07 10:02:52 +08:00
shenjian@fangjia.com 2a75739658 修复美团csv问题 2023-04-07 09:57:46 +08:00
shenjian@fangjia.com e66e8e75ce 修复美团csv问题 2023-04-07 09:50:25 +08:00
shenjian@fangjia.com c48580c31e 修复美团 2023-04-06 17:50:14 +08:00
shenjian@fangjia.com feecd858fa 修复美团 2023-04-06 17:48:05 +08:00
shenjian@fangjia.com 5107701487 修复美团 2023-04-06 17:35:44 +08:00
shenjian@fangjia.com f082ff8739 修复美团类型问题 2023-04-06 16:48:00 +08:00
shenjian@fangjia.com c8537800a8 修复美团类型问题 2023-04-06 16:23:02 +08:00
shenjian@fangjia.com 0ee261a545 修复美团m_date报错 2023-04-06 15:03:49 +08:00
shenjian@fangjia.com 9f379f106c 修复美团 2023-04-06 14:37:12 +08:00
shenjian@fangjia.com 4008f7937f 异常捕获 2023-04-06 14:22:07 +08:00
shenjian@fangjia.com e39c52849c 优化 2023-04-06 13:47:44 +08:00
shenjian@fangjia.com 8f170e86fa 优化 2023-03-31 14:34:12 +08:00
shenjian@fangjia.com 6488f4a40a 优化 2023-03-31 14:30:01 +08:00
shenjian@fangjia.com 2666137af9 新增置信度 2023-03-31 14:27:10 +08:00
shenjian@fangjia.com a5982ecfa0 新增置信度 2023-03-31 13:10:48 +08:00
shenjian@fangjia.com bf7b1174f3 新增置信度 2023-03-31 11:46:30 +08:00
shenjian@fangjia.com b7ab8b9192 新增置信度 2023-03-29 17:01:13 +08:00
shenjian@fangjia.com feab049058 新增置信度 2023-03-29 16:51:15 +08:00
shenjian@fangjia.com 3f9e8a7528 优化租金均价覆盖率统计 2023-03-29 16:43:38 +08:00
shenjian@fangjia.com 2eecb42410 新增字段统计 2023-03-29 15:55:48 +08:00
shenjian@fangjia.com 3ab1d31852 新增字段统计 2023-03-29 15:20:32 +08:00
shenjian@fangjia.com cf9c99a416 新增字段统计 2023-03-29 15:15:24 +08:00
shenjian@fangjia.com 98017d34f0 新增字段统计 2023-03-29 15:14:02 +08:00
shenjian@fangjia.com 9abc7a5676 物业费添加分数 2023-03-29 13:25:50 +08:00
shenjian@fangjia.com d43a950e83 物业费添加分数 2023-03-29 13:22:09 +08:00
shenjian@fangjia.com e0e18e5d36 物业费添加分数 2023-03-29 11:48:02 +08:00
shenjian@fangjia.com b41c78fef4 物业费添加分数 2023-03-29 11:25:24 +08:00
shenjian@fangjia.com b7c567b2fb 户均总价补充方式修改及添加分数 2023-03-29 11:02:50 +08:00
shenjian@fangjia.com 16b812967c 优化估值检测 2023-03-28 15:23:08 +08:00
shenjian@fangjia.com e8c124a1e9 优化估值检测 2023-03-28 15:12:51 +08:00
shenjian@fangjia.com 2373b2f879 优化估值检测 2023-03-28 15:09:50 +08:00
shenjian@fangjia.com 7838d54e8c 优化估值检测 2023-03-28 15:02:22 +08:00
shenjian@fangjia.com 0fae0996ef 优化估值检测 2023-03-28 15:00:51 +08:00
shenjian@fangjia.com 27e7f3fae6 优化估值检测 2023-03-28 15:00:12 +08:00
shenjian@fangjia.com bb9981a5d9 优化估值检测 2023-03-28 14:55:14 +08:00
shenjian@fangjia.com b5da624d93 优化估值检测 2023-03-28 14:53:40 +08:00
shenjian@fangjia.com 02bcba8a29 新增具体责任人 2023-03-27 10:47:14 +08:00
shenjian@fangjia.com 0727376478 优化相对价格排名 2023-03-23 09:47:24 +08:00
shenjian@fangjia.com 29d42f33d6 优化相对价格排名 2023-03-23 09:25:19 +08:00
shenjian@fangjia.com 3d2acbddaa 优化统计 2023-03-22 18:20:38 +08:00
shenjian@fangjia.com 65d128dde2 优化统计 2023-03-22 18:16:33 +08:00
shenjian@fangjia.com 8c439e3201 优化统计 2023-03-22 18:15:17 +08:00
shenjian@fangjia.com 0fe9d8f99a 优化统计 2023-03-22 17:56:00 +08:00
shenjian@fangjia.com ea6ae494cd 相对价格排名打分 2023-03-22 10:15:30 +08:00
shenjian@fangjia.com 85669b6bd7 优化pgsql 2023-03-21 11:43:09 +08:00
shenjian@fangjia.com a2fa2be77a 优化 2023-03-21 11:07:07 +08:00
gaojuan 650e7e0795 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2023-03-20 18:28:19 +08:00
gaojuan 940f7d85e9 update 2023-03-20 18:28:08 +08:00
shenjian@fangjia.com ac42eb2812 新增因子不足清理 2023-03-20 18:04:11 +08:00
shenjian@fangjia.com 109db091e0 新增因子不足清理 2023-03-20 17:59:23 +08:00
gaojuan ba51ae5ee3 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2023-03-20 17:44:42 +08:00
shenjian@fangjia.com 713e358000 修复别墅物业费bug 2023-03-20 17:14:38 +08:00
shenjian@fangjia.com db22cb2cb5 修复物业费bug 2023-03-20 16:34:31 +08:00
shenjian@fangjia.com 7c99f1b0ac 美团中间表改造 2023-03-20 13:19:14 +08:00
shenjian@fangjia.com 11d76204a7 优化评级 2023-03-20 10:09:57 +08:00
Jian Shen caf18c2d8a 测试触发器 2023-03-18 20:01:48 +08:00
Jian Shen da08f4b06c 测试触发器 2023-03-18 15:23:50 +08:00
Jian Shen 0cfe28ae48 测试触发器 2023-03-18 15:17:57 +08:00
Jian Shen 0876b47a47 测试触发器 2023-03-18 14:34:47 +08:00
Jian Shen 15bc05797e 测试触发器 2023-03-18 14:32:49 +08:00
Jian Shen e04026491d 测试触发器 2023-03-18 14:30:51 +08:00
Jian Shen 891419b45a 测试触发器 2023-03-18 14:28:01 +08:00
Jian Shen d8543eae5b 优化写入中间表 2023-03-18 14:14:27 +08:00
Jian Shen 8406955a7e 优化写入中间表 2023-03-18 13:34:31 +08:00
Jian Shen 2654fcfe67 优化写入中间表 2023-03-18 13:13:58 +08:00
Jian Shen 94cf7da75d 优化安装 2023-03-18 12:21:21 +08:00
Jian Shen 94f2b602dc worker重启bug 2023-03-18 11:12:33 +08:00
Jian Shen cf60427ef0 worker重启bug 2023-03-17 23:05:30 +08:00
Jian Shen 034a2a80ba 修改日期 2023-03-17 21:01:37 +08:00
Jian Shen e5172f0d6c 修改日期 2023-03-17 20:39:39 +08:00
Jian Shen c31e32598f 修改日期 2023-03-17 20:37:30 +08:00
shenjian@fangjia.com a1c65a7633 优化下载 2023-03-17 17:46:26 +08:00
shenjian@fangjia.com 50cbe55556 暂时不下载 2023-03-17 15:38:16 +08:00
shenjian@fangjia.com b8e8d79821 优化中间表 2023-03-17 12:21:22 +08:00
shenjian@fangjia.com fe592e20ac 优化中间表 2023-03-17 11:18:37 +08:00
shenjian@fangjia.com 3572add969 init 2023-03-16 14:57:42 +08:00
shenjian@fangjia.com f43f99bfaf init 2023-03-16 14:33:16 +08:00
shenjian@fangjia.com 70e5bbaf28 init 2023-03-16 14:31:58 +08:00
shenjian@fangjia.com e2573a5f1c airflow ignore 2023-03-16 14:04:35 +08:00
shenjian@fangjia.com 0767704ed0 airflow ignore 2023-03-15 10:53:11 +08:00
shenjian@fangjia.com 9df55a9883 airflow ignore 2023-03-15 10:39:50 +08:00
shenjian@fangjia.com 49cb6a3023 airflow ignore 2023-03-15 10:30:26 +08:00
shenjian@fangjia.com 60de058719 airflow ignore 2023-03-15 10:24:31 +08:00
shenjian@fangjia.com e8bca1f2cb airflow ignore 2023-03-15 10:21:32 +08:00
shenjian@fangjia.com 37997feb6e airflow ignore 2023-03-15 10:10:22 +08:00
shenjian@fangjia.com 202111ef0f 优化ignore 2023-03-15 09:45:30 +08:00
shenjian@fangjia.com 1c241ef841 优化 2023-03-15 09:34:59 +08:00
shenjian@fangjia.com 15453e103c 优化 2023-03-14 17:43:32 +08:00
shenjian@fangjia.com 999f98e088 第一个任务install 2023-03-14 17:09:01 +08:00
shenjian@fangjia.com c4e801d537 优化物业费入模 2023-03-14 17:04:34 +08:00
shenjian@fangjia.com 6c4385b4b4 优化物业费入模 2023-03-14 17:01:16 +08:00
shenjian@fangjia.com c2455331a6 优化物业费 2023-03-14 16:08:52 +08:00
shenjian@fangjia.com 3e23948677 安装依赖放置在第一个任务中,便于日志查看 2023-03-14 14:10:34 +08:00
shenjian@fangjia.com ea796a823b 安装依赖放置在第一个任务中,便于日志查看 2023-03-14 14:05:17 +08:00
shenjian@fangjia.com 08950f621f 安装依赖放置在第一个任务中,便于日志查看 2023-03-14 13:41:53 +08:00
shenjian@fangjia.com 78f210db5d 安装依赖放置在第一个任务中,便于日志查看 2023-03-14 13:36:13 +08:00
shenjian@fangjia.com c0fcafc2fc 新增 2023-03-14 13:22:32 +08:00
shenjian@fangjia.com 6821a74975 新增 2023-03-14 13:18:09 +08:00
shenjian@fangjia.com 25c2972c54 优化导入 2023-03-14 13:10:09 +08:00
shenjian@fangjia.com 652c48b73c 优化导入 2023-03-14 12:23:17 +08:00
shenjian@fangjia.com db24839786 优化导入 2023-03-14 12:13:07 +08:00
shenjian@fangjia.com 15ac2a3501 优化导入 2023-03-14 11:47:05 +08:00
shenjian@fangjia.com 5d31462b43 优化导入 2023-03-14 11:43:55 +08:00
shenjian@fangjia.com dda9c85c48 优化导入 2023-03-14 11:22:09 +08:00
shenjian@fangjia.com 5b15017380 优化导入 2023-03-14 11:13:25 +08:00
shenjian@fangjia.com 4db4e6b19d 优化导入 2023-03-14 11:00:32 +08:00
shenjian@fangjia.com 6b2728d693 优化融合检测 2023-03-14 09:48:26 +08:00
shenjian@fangjia.com d327ad1324 优化融合检测 2023-03-14 09:46:19 +08:00
shenjian@fangjia.com 071990bf85 新增父类 2023-03-13 10:03:52 +08:00
Jian Shen 7b6c28794e 优化动态安装 2023-03-11 18:00:20 +08:00
Jian Shen 449076a561 优化动态安装 2023-03-11 18:00:05 +08:00
Jian Shen abf65edf14 优化动态安装 2023-03-11 17:57:10 +08:00
Jian Shen f8cfc224ae 优化动态安装 2023-03-11 17:55:14 +08:00
Jian Shen a286e5e8af 优化动态安装 2023-03-11 17:50:04 +08:00
Jian Shen 128eeb47e9 优化动态安装 2023-03-11 17:45:40 +08:00
Jian Shen f2dfc4da85 优化动态安装 2023-03-11 17:41:45 +08:00
Jian Shen d201872c66 优化动态安装 2023-03-11 17:29:50 +08:00
Jian Shen c8093fce87 优化动态安装 2023-03-11 17:27:16 +08:00
Jian Shen e581409541 优化动态安装 2023-03-11 17:03:40 +08:00
Jian Shen 34b25bdfd2 优化动态安装 2023-03-11 16:47:23 +08:00
Jian Shen 15acb1eb5c 优化动态安装 2023-03-11 16:46:09 +08:00
Jian Shen b747587402 优化动态安装 2023-03-11 16:41:19 +08:00
Jian Shen 3b3ac26094 优化动态安装 2023-03-11 15:40:50 +08:00
Jian Shen dea4e5b23d 优化动态安装 2023-03-11 15:29:16 +08:00
Jian Shen 3cff0c88d4 优化动态安装 2023-03-11 15:15:53 +08:00
Jian Shen b2abcf1200 优化动态安装 2023-03-11 15:11:09 +08:00
Jian Shen 63b6d49dac 优化动态安装 2023-03-11 15:10:13 +08:00
Jian Shen f707184d93 Merge remote-tracking branch 'origin/master'
# Conflicts:
#	dags/init_dag.py
2023-03-11 13:32:20 +08:00
Jian Shen e065f8632b 优化动态安装 2023-03-11 13:32:08 +08:00
Jian Shen 8345545b0e 优化动态安装 2023-03-11 13:29:19 +08:00
Jian Shen 568a06fe0b Merge remote-tracking branch 'origin/master' 2023-03-11 13:15:29 +08:00
Jian Shen 5cac0cd831 优化动态安装 2023-03-11 13:15:20 +08:00
Jian Shen 1dc4d674f4 优化动态安装 2023-03-11 13:10:23 +08:00
Jian Shen 28096a64b8 优化动态安装 2023-03-11 13:07:18 +08:00
Jian Shen 0a98c28ba3 优化动态安装 2023-03-11 12:40:38 +08:00
Jian Shen b4056ef74a 优化动态安装 2023-03-11 10:34:27 +08:00
Jian Shen 84607517db 优化动态安装 2023-03-11 10:26:14 +08:00
Jian Shen e5e699c8da 优化动态安装 2023-03-11 10:25:13 +08:00
Jian Shen 0b78d4c5ce Merge remote-tracking branch 'origin/master' 2023-03-11 10:24:50 +08:00
Jian Shen bcf64ccca1 优化动态安装 2023-03-11 10:24:38 +08:00
shenjian@fangjia.com 71d00d28dd 优化相对价格排名性能 2023-03-10 17:37:21 +08:00
shenjian@fangjia.com c9e51dd426 人均消费区域级别极值过滤 2023-03-10 17:07:51 +08:00
shenjian@fangjia.com 98466f4b22 户均总价补全调整 2023-03-10 13:16:20 +08:00
shenjian@fangjia.com 437bb336bd 新增城市范围圈定 2023-03-10 09:30:13 +08:00
shenjian@fangjia.com ccc566f6d1 物业费噪音词前置处理 2023-03-09 18:26:04 +08:00
shenjian@fangjia.com 261270221b 物业费噪音词前置处理 2023-03-09 17:37:33 +08:00
shenjian@fangjia.com 7aa9778413 加锁 2023-03-09 17:24:10 +08:00
shenjian@fangjia.com e90b282e73 注释取消 2023-03-09 17:05:11 +08:00
shenjian@fangjia.com 44597fd073 新增锁 2023-03-09 16:58:26 +08:00
shenjian@fangjia.com 0466cd3772 多线程问题排查 2023-03-09 16:53:22 +08:00
shenjian@fangjia.com b131fa673e 多线程问题排查 2023-03-09 16:52:14 +08:00
shenjian@fangjia.com 2fa43377fe 多线程问题排查 2023-03-09 16:51:36 +08:00
shenjian@fangjia.com f31502935d 多线程问题排查 2023-03-09 16:40:07 +08:00
shenjian@fangjia.com 32798ccefc 调试 2023-03-09 16:36:04 +08:00
shenjian@fangjia.com 09a2733f8e 调试 2023-03-09 16:32:02 +08:00
shenjian@fangjia.com 48ceec2760 调试 2023-03-09 16:28:43 +08:00
shenjian@fangjia.com 6688e93b99 调试 2023-03-09 16:06:55 +08:00
shenjian@fangjia.com 57e5a5e649 Merge remote-tracking branch 'origin/master' 2023-03-09 15:56:41 +08:00
shenjian@fangjia.com a76f9b14a2 调试 2023-03-09 15:56:34 +08:00
shenjian@fangjia.com b43c7857c2 调试 2023-03-09 15:56:23 +08:00
shenjian@fangjia.com 484c65ce15 调试 2023-03-09 15:55:33 +08:00
shenjian@fangjia.com ed57487aef 调试 2023-03-09 15:51:23 +08:00
shenjian@fangjia.com 42d78f95e5 调试 2023-03-09 15:48:45 +08:00
shenjian@fangjia.com 9245b3fe96 调试 2023-03-09 15:44:54 +08:00
shenjian@fangjia.com 8d37a92cf6 调试 2023-03-09 15:37:18 +08:00
shenjian@fangjia.com 4d6145fb89 调试 2023-03-09 15:25:05 +08:00
shenjian@fangjia.com 9e7d8d1743 修改线上null 2023-03-09 15:22:37 +08:00
shenjian@fangjia.com 25d046cc6e 修改线上null 2023-03-09 15:09:42 +08:00
shenjian@fangjia.com 44cee71ce3 编排修改 2023-03-09 15:01:20 +08:00
shenjian@fangjia.com c9e802da90 降低并行线程数 2023-03-09 14:38:00 +08:00
shenjian@fangjia.com 68e154d390 人均消费价格选取策略修改 2023-03-09 13:40:31 +08:00
shenjian@fangjia.com 9e07fdec9a 财力评级人均消费噪音词处理 2023-03-09 12:25:58 +08:00
shenjian@fangjia.com 66a623a060 任务并行编排 2023-03-07 15:15:42 +08:00
shenjian@fangjia.com 00c76af03a 任务并行编排 2023-03-07 15:14:11 +08:00
shenjian@fangjia.com a423fe10c1 任务并行编排 2023-03-07 15:05:46 +08:00
shenjian@fangjia.com 6ce1606b51 并发数降低 2023-03-07 14:53:15 +08:00
shenjian@fangjia.com 1ad3654416 Merge remote-tracking branch 'origin/master' 2023-03-07 14:46:29 +08:00
shenjian@fangjia.com 0cc1670467 修改不更新bug 2023-03-07 14:36:19 +08:00
shenjian@fangjia.com 087ef47d4e 修改不更新bug 2023-03-07 14:32:06 +08:00
shenjian@fangjia.com afefe99b0f 修改不更新bug 2023-03-07 14:31:47 +08:00
shenjian@fangjia.com b7d4180cde 人均消费提高最大线程 2023-03-07 14:21:06 +08:00
shenjian@fangjia.com 0ded5b6901 修复人均消费bug 2023-03-07 14:03:55 +08:00
shenjian@fangjia.com 05ee877888 暂时注释 2023-03-06 16:49:40 +08:00
shenjian@fangjia.com 98266e9c2b 优化人均消费bug 2023-03-06 15:49:30 +08:00
shenjian@fangjia.com f746d854c7 优化人均消费bug 2023-03-06 15:46:03 +08:00
shenjian@fangjia.com b00f3a67a1 新增人口规模 2023-03-06 15:28:19 +08:00
shenjian@fangjia.com 4d6c59d043 户均总价计算逻辑调整 2023-03-03 16:56:17 +08:00
shenjian@fangjia.com 87e931f200 新增近三个月挂牌量 2023-02-27 10:33:09 +08:00
shenjian@fangjia.com acc58f437a Merge remote-tracking branch 'origin/master' 2023-02-27 10:30:24 +08:00
shenjian@fangjia.com 37aa051bf9 优化统计 2023-02-27 10:30:08 +08:00
admin 010a9db93b Merge branch 'xuhe' into 'master'
估值监测和波动变化修改数据库连接

See merge request !4
2023-02-25 10:34:33 +08:00
admin ffe2f5fcbd Merge branch 'xuhe' into 'master'
Xuhe

See merge request !3
2023-02-24 18:29:56 +08:00
gaojuan c6ef25d6b4 Merge branch 'master' of http://gitlab.fangjia.com/dev0/airflow 2022-11-21 16:40:05 +08:00
gaojuan c0b9224679 修改时间 2022-11-21 16:39:56 +08:00
175 changed files with 17859 additions and 1328 deletions
-6
View File
@@ -1,6 +0,0 @@
poi/business/.*
stat/evaluate/*
stat/community/*
mt/supply/*
business/business/*
plugin/**
+2 -1
View File
@@ -7,4 +7,5 @@ pip install -r requirements.txt -i https://repo.fangjia.com/repository/pypi-publ
- 具体业务逻辑放入xxx/business中,如poi/business - 具体业务逻辑放入xxx/business中,如poi/business
- 添加.airflowignore xxx/business/*, 如poi/business/* - 添加.airflowignore xxx/business/*, 如poi/business/*
- dag放在xxx目录下即可,命名为xx_dag.py - dag放在xxx目录下即可,命名为dag_xx.py,之前命名为xx_dag导致xx与报名相同而加入airflowignore后dag不显示问题,
如merge_dag会不显示,而dag_merge即可,怀疑是jh/merge/*的bug
+25
View File
@@ -0,0 +1,25 @@
poi/business/*
stat/evaluate/*
stat/community/*
stats/evaluate/*
stats/community/*
stats/aiprice/*
stats/cityPrice/*
stats/test/*
mt/supply/*
business/business/*
jh/merge/*
jh/para/*
jh/price/*
jh/sample/*
db/*
office/rent/*
office/sell/*
shop/rent/*
shop/sell/*
company/company/*
ai/stock/*
price_exponents/region_price_month.py
price_exponents/seaweed_price_month.py
test/*
train/*
+23
View File
@@ -0,0 +1,23 @@
# from airflow.providers.standard.operators.bash import BashOperator
# import pendulum
# from airflow import DAG
#
# with DAG(
# dag_id="ai_stock_dag",
# # 每天运行
# schedule="30 8 * * 1-5",
# start_date=pendulum.datetime(2025, 11, 10, tz="Asia/Shanghai"),
# catchup=True,
# tags=["ai_stock"],
# ) as dag:
#
# t1 = BashOperator(
# task_id='stock_strategy',
# bash_command='python /opt/airflow/dags/repo/dags/ai/stock/stock_strategy.py',
# )
#
# t2 = BashOperator(
# task_id='ai_stock',
# bash_command='python /opt/airflow/dags/repo/dags/ai/stock/stock.py',
# )
# t1 >> t2
+38
View File
@@ -0,0 +1,38 @@
# from goojia_utils import MysqlClient, ArkAIClient, dingutils
# from goojia_utils import dateutils
# from datetime import datetime
# import requests
#
# db_util = MysqlClient(
# host="192.168.0.93",
# port=3306,
# database="stock",
# user="root",
# password="Goojia123456"
# )
# date = int(dateutils.format_date(dateutils.add_months(datetime.today(), months=-1), fm='%Y%m%d'))
# select_stocks = db_util.scan_by_sql_to_data_frame("select ts_code as 股票代码, stock_name as 股票名称,close as 股价, pe as PE,turnover_rate as 换手率 from mu_stocks_new where `trade_date`= (select max(`trade_date`) from mu_stocks_new)")
# stat = db_util.scan_by_sql_to_data_frame(f"select ts_code as 股票代码, count(ts_code) as 命中次数 from mu_stocks_new where `trade_date` > {date} group by ts_code")
# select_stocks = select_stocks.merge(stat, on='股票代码', how='left')
# select_stocks['命中次数'] = select_stocks['命中次数'].fillna(0)
# select_stocks = select_stocks.sort_values(by=['命中次数', 'PE'], ascending=[False, True])
# def df_to_nested_list(df):
# """转换为嵌套的 Markdown 列表"""
# lines = []
# i= 1
# for _, row in df.iterrows():
# lines.append(f"{i}. **{row['股票名称']}** ({row['股票代码']}) PE: {row['PE']} 股价: {row['股价']:.2f} 出现次数: {row['命中次数']} 换手率: {row['换手率']:.2f}")
# i = i + 1
# return '\n'.join(lines)
# data = {
# "msgtype": "markdown",
# "markdown": {
# "title": dateutils.format_date(datetime.today(), fm='%Y%m%d') + "选股策略播报",
# "text": df_to_nested_list(select_stocks)
# }
# }
# access_token = "1e5f9aaeb90b091bc79652b52800cad2f1dee1ac0990149252511965f6d1c12a"
# # 发送消息至群
# msg = requests.post(url='https://oapi.dingtalk.com/robot/send?access_token=' + access_token, json=data).json()
# msg = requests.post(url='https://oapi.dingtalk.com/robot/send?access_token=45ac47a34dc5491535d77d643512244a85d91d7e74676e9c42446dc06357326e', json=data).json()
# print(msg)
+100
View File
@@ -0,0 +1,100 @@
import datetime
from sqlalchemy import create_engine
from dateutil.relativedelta import relativedelta
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.mixture import GaussianMixture
import numpy as np
from collections import defaultdict
engine = create_engine('mysql+pymysql://root:Goojia123456@192.168.0.93:3306/stock')
mu_stocks_all = pd.read_sql(f"select * from mu_stocks_new", con = engine)
stock_basic_infos = pd.read_sql(f"select * from stock_basic", con=engine)
trade_dates = mu_stocks_all['trade_date'].tolist()
today = datetime.datetime.now()
def gmm_trade(get_data,df):
gmm = GaussianMixture(n_components=100, covariance_type='spherical', random_state=0)
data = df.values
data = np.array(data, dtype='float64')
x = data
gmm.fit(x)
y_hat = gmm.predict(x)
gmm_means_ = [i[0] for i in gmm.means_]
d = defaultdict(list)
for k, va in [(v, i) for i, v in enumerate(gmm_means_)]:
d[va] = k
gmm_means_list = []
for label in y_hat:
mean_value = d[label]
gmm_means_list.append(mean_value)
get_data["rating"] = y_hat
get_data["mean"] = gmm_means_list
return get_data
def stock_gmm(last_day_str,last_month_str):
start = datetime.datetime.now()
stock_all = pd.read_sql(f"select * from stock_index_daily where trade_date>'{last_month_str}' and trade_date<='{last_day_str}'", con = engine)
end = datetime.datetime.now()
print('MySQL Query Running time: %s Seconds' % (end - start))
stock_all = stock_all.sort_values(by="trade_date", ascending=False)
stock_select = stock_all[((stock_all['ts_code'].str.startswith('600')) | (stock_all['ts_code'].str.startswith('00')))]
stock_data = stock_select.dropna(subset=['turnover_rate_f', 'volume_ratio', 'pe', 'pe_ttm', 'pb', 'ps_ttm', 'ps'])
stock_group = stock_data.groupby(['ts_code'])
scaler = StandardScaler()
import_col = ['turnover_rate_f', 'volume_ratio', 'pe', 'pe_ttm', 'pb', 'ps_ttm', 'ps']
scaler_col = [str(i) for i in range(len(import_col))]
res = []
for ts_code, ts_detail in stock_group:
if len(ts_detail) > 3:
ts_detail[scaler_col] = ts_detail.apply(lambda x: (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0), axis=1,
result_type="expand")
ts_detail[scaler_col] = scaler.fit_transform(ts_detail[import_col].values)
for i, row in ts_detail.iterrows():
res.append(row)
stock_res = pd.DataFrame(res)
stock_res.to_excel('E:\\工作文件\\avm_output\\airflow_scaled_df.xlsx', encoding='utf-8', index=False)
stock_all_newest = stock_all[stock_all.trade_date == stock_all.trade_date.max()]
stock_all_newest = stock_all_newest[['ts_code', 'turnover_rate', 'close', 'amount', 'pe', 'pe_ttm', 'total_mv', 'pct_chg']]
X = stock_res[['0', '1', '2', '3', '4', '5', '6']]
gmm_df = gmm_trade(stock_res, X)
gmm_df.to_excel('E:\\工作文件\\avm_output\\airflow_gmm_df.xlsx', encoding='utf-8', index=False)
gmm_group_df = gmm_df.groupby(['ts_code', 'rating'])['stock_name'].count().reset_index()
gmm_group_df.rename(columns={'stock_name': 'count'}, inplace=True)
gmm_df = gmm_df.merge(gmm_group_df, how='left', on=['ts_code', 'rating'])
min_mean = gmm_df['mean'].min()
slecct_df = gmm_df[gmm_df['mean'] == min_mean]
count_max = slecct_df['count'].max()
slecct_df = slecct_df[slecct_df['count'] == count_max]
slecct_df.drop_duplicates(subset=["ts_code"], keep="first", inplace=True)
block_list = list(slecct_df['stock_name'])
select_block = []
for name in block_list:
status = 1
for i in ['N', 'ST']:
if i in name:
status = 0
if status == 1:
select_block.append(name)
return stock_all_newest,select_block,slecct_df
def stock_select(i):
last_day = today+relativedelta(days=-i)
last_month = last_day+relativedelta(days=-15)
last_day_str = last_day.strftime('%Y%m%d')
last_day_str = '20260325'
# if last_day_str in trade_dates:
# return last_day_str
# if int(last_day_str)<20200101:
# return last_day_str
last_month_str = last_month.strftime('%Y%m%d')
last_month_str = '20260304'
stock_all_newest,select_block,slecct_df=stock_gmm(last_day_str,last_month_str)
db_stock = slecct_df[['ts_code', 'stock_name', 'open', 'close', 'high', 'low',
'vol', 'amount', 'pre_close', 'change', 'pct_chg', 'turnover_rate',
'turnover_rate_f', 'volume_ratio', 'pe', 'pe_ttm', 'pb', 'ps_ttm', 'ps',
'total_share', 'float_share', 'free_share', 'total_mv', 'circ_mv']].copy()
db_stock['trade_date'] = last_day_str
print(select_block)
# db_stock['ts_code'] = db_stock['ts_code'].apply(lambda x: x[0:6])
# db_stock.to_sql("mu_stocks_new",con=engine, index=False, if_exists="append")
return last_day_str
if __name__ == '__main__':
stock_select(0)
+208
View File
@@ -0,0 +1,208 @@
# -*- coding: utf-8 -*-
import os
import datetime
import calendar
import json
import pandas as pd
from goojia_utils import LOG_MONGO_CLIENT
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.email import send_email
import pendulum
def get_last_month_range(ref_date: datetime.date = None):
"""
获取参考日期的上个月第一天 00:00:00 与最后一天 23:59:59.999999
以北京时间为准
"""
if ref_date is None:
ref_date = datetime.date.today()
first_day_current_month = ref_date.replace(day=1)
last_day_last_month = first_day_current_month - datetime.timedelta(days=1)
first_day_last_month = last_day_last_month.replace(day=1)
start_time = datetime.datetime.combine(first_day_last_month, datetime.time.min)
end_time = datetime.datetime.combine(last_day_last_month, datetime.time.max)
return start_time, end_time, first_day_last_month.year, first_day_last_month.month
def parse_dict(val):
"""将字符串类型的 JSON 或字典解析为 dict"""
if isinstance(val, dict):
return val
if isinstance(val, str) and val.strip():
try:
return json.loads(val)
except Exception:
return {}
return {}
# Excel 中文表头定义:区分【输入】和【输出】
EXCEL_COLUMNS = [
'请求时间',
# 输入参数
'输入-城市',
'输入-地址',
# 响应参数 (取自 res_map.result)
'输出-省份',
'输出-省份编码',
'输出-城市名称',
'输出-城市编码',
'输出-区域',
'输出-区域编码',
'输出-乡镇街道',
'输出-乡镇街道编码',
'输出-是否涉农',
'输出-所属区域',
'输出-地址验证',
'输出-是否为县级市',
'输出-是否县城城区',
'输出-是否为省国家级开发区',
'输出-省国家级开发区名称',
'输出-开发区等级'
]
def export_and_send_njcb_statement(**context):
"""
查询上月对公涉农API日志,生成全中文表头Excel对账单并通过邮件发送给顾老师
"""
now = datetime.datetime.now()
start_time, end_time, last_year, last_month = get_last_month_range(now.date())
print(f"统计时间范围: {start_time} 至 {end_time} (归属年月: {last_year}年{last_month:02d}月)")
open_uri_log_conn = LOG_MONGO_CLIENT.get_collection('business', 'open_uri_log')
# 查询条件:
# user_id = 1007801, request_uri = /addrSvc/njcbVillageIdentify, c_date在时间范围内, 且只输出code是200的响应
query = {
'user_id': '1007801',
'request_uri': '/addrSvc/njcbVillageIdentify',
'c_date': {
'$gte': start_time,
'$lte': end_time
},
'$or': [
{'code': 200},
{'code': '200'}
]
}
print(f"开始查询 MongoDB business.open_uri_log,条件: {query}")
cursor = open_uri_log_conn.find(query).batch_size(2000)
rows = []
scanned_count = 0
for doc in cursor:
scanned_count += 1
param_map = parse_dict(doc.get('parameter_map'))
res_map = parse_dict(doc.get('res_map'))
# res_map 结构中有 code, msg, result 嵌套
# 实际业务返回字段在 res_map['result'] 中,若无 result 则回退直接使用 res_map
result_map = parse_dict(res_map.get('result')) if res_map.get('result') else res_map
# 过滤只保留"对公涉农"的请求
# 涉农类型: 1 对私,2 对公 (默认为对私)
req_type = str(param_map.get('type', '1')).strip()
if req_type != '2':
continue
# 仅输出"输出-是否涉农"为"农"的部分
verify_flag = str(result_map.get('verifyFlag', '')).strip()
if verify_flag != '农':
continue
c_date = doc.get('c_date')
if isinstance(c_date, datetime.datetime):
c_date_str = c_date.strftime('%Y-%m-%d %H:%M:%S')
else:
c_date_str = str(c_date) if c_date is not None else ''
row = {
'请求时间': c_date_str,
# 输入参数
'输入-城市': param_map.get('city', ''),
'输入-地址': param_map.get('address', ''),
# 响应参数 (取自 result_map)
'输出-省份': result_map.get('province', ''),
'输出-省份编码': result_map.get('provinceCode', ''),
'输出-城市名称': result_map.get('city', ''),
'输出-城市编码': result_map.get('cityCode', ''),
'输出-区域': result_map.get('district', ''),
'输出-区域编码': result_map.get('districtCode', ''),
'输出-乡镇街道': result_map.get('town', ''),
'输出-乡镇街道编码': result_map.get('townCode', ''),
'输出-是否涉农': verify_flag,
'输出-所属区域': result_map.get('addrArea', ''),
'输出-地址验证': result_map.get('addrVerify', ''),
'输出-是否为县级市': result_map.get('countyLevelCity', ''),
'输出-是否县城城区': result_map.get('countyGov', ''),
'输出-是否为省国家级开发区': result_map.get('devDistrict', ''),
'输出-省国家级开发区名称': result_map.get('devDistrictName', ''),
'输出-开发区等级': result_map.get('devLevel', '')
}
rows.append(row)
total_count = len(rows)
print(f"扫描命中记录: {scanned_count} 条,其中符合对公且是否涉农为'农'的记录: {total_count} 条")
# 生成 Excel 文件
tmp_dir = os.path.join(os.path.expanduser('~'), 'tmp', 'njcb_statement')
os.makedirs(tmp_dir, exist_ok=True)
filename = f"南京银行涉农API对公涉农调用流水_{last_year}{last_month:02d}.xlsx"
excel_path = os.path.join(tmp_dir, filename)
if rows:
df = pd.DataFrame(rows, columns=EXCEL_COLUMNS)
else:
df = pd.DataFrame(columns=EXCEL_COLUMNS)
df.to_excel(excel_path, index=False, engine='openpyxl')
print(f"Excel文件生成成功: {excel_path}")
# 发送邮件
receiver = 'guyzh@njcb.com.cn'
cc_receiver = ['wangwenyan@fangjia.com', 'daijinshui@fangjia.com']
subject = f"南京银行{last_year}年{last_month:02d}月涉农API对公涉农调用流水"
html_content = f"""
<h3>顾老师:</h3>
<p>您好!</p>
<p>附件为南京银行 <b>{last_year}年{last_month:02d}月</b> 涉农识别API(/addrSvc/njcbVillageIdentify)对公涉农调用流水,请查收。</p>
<ul>
<li><b>对账周期:</b> {start_time.strftime('%Y-%m-%d')} 至 {end_time.strftime('%Y-%m-%d')}</li>
<li><b>调用有效成功条数(Code=200 且 对公且判定为农):</b> {total_count} 条</li>
</ul>
<p>如有任何疑问,请随时与我们联系。</p>
"""
print(f"正在发送邮件至: {receiver}, 抄送: {cc_receiver}, 附件: {excel_path}")
send_email(
to=receiver,
subject=subject,
html_content=html_content,
files=[excel_path],
cc=cc_receiver
)
print("邮件发送完成。")
# Airflow DAG 定义:每月2号上午9:00执行(北京时间)
with DAG(
dag_id="njcb_village_identify_statement_dag",
schedule="0 9 2 * *",
start_date=pendulum.datetime(2024, 1, 1, tz="Asia/Shanghai"),
catchup=False,
description="每月2号给南京银行顾老师发送上个月涉农API对公涉农调用流水",
tags=["api", "njcb", "statement"]
) as dag:
send_statement_task = PythonOperator(
task_id="send_njcb_village_statement",
python_callable=export_and_send_njcb_statement
)
+22 -17
View File
@@ -1,27 +1,32 @@
from goojia_utils import REDIS_CLIENT from goojia_utils import REDIS_CLIENT
import json import json
import logging
log = logging.getLogger(__name__)
""" """
高德key每日定时生成 高德key每日定时生成
""" """
#amap_key = {"4e9592208336433ee1b7aa4d1d434418", "10bcb421d2ee4e51ed9268d6dca58ad1"} #amap_key = {"4e9592208336433ee1b7aa4d1d434418", "10bcb421d2ee4e51ed9268d6dca58ad1"}
log.info("测试生成key")
#老板新注册key,限制15万 #老板新注册key,限制15万
amap_key = {"6e7b0b0539ff51a714678cd69183fe32"} # amap_key = {"6e7b0b0539ff51a714678cd69183fe32"}
#
service = {'v3/geocode/geo': 150000, # service = {'v3/geocode/geo': 150000,
'v3/geocode/regeo': 150000, # 'v3/geocode/regeo': 150000,
'v3/place/text': 150000, # 'v3/place/text': 150000,
'v5/place/text': 150000, # 'v5/place/text': 150000,
'v3/place/around': 150000, # 'v3/place/around': 150000,
'v5/place/around': 150000, # 'v5/place/around': 150000,
'v3/place/detail': 150000, # 'v3/place/detail': 150000,
'v5/place/detail': 150000} # 'v5/place/detail': 150000}
#
for service_key in service.keys(): # for service_key in service.keys():
val = dict() # val = dict()
for key in amap_key: # for key in amap_key:
val[key] = service[service_key] # val[key] = service[service_key]
#
REDIS_CLIENT.hset("amapKey", service_key, json.dumps(val)) # REDIS_CLIENT.hset("amapKey", service_key, json.dumps(val))
+85
View File
@@ -0,0 +1,85 @@
from goojia_utils import LOG_MONGO_CLIENT,dingutils
from sqlalchemy import create_engine
import datetime
import pandas as pd
# mysql连接
engine = create_engine('mysql+pymysql://sa:G00jia@192.168.0.42:4306/fangjia?charset=utf8')
# mongodb连接
open_uri_log_conn = LOG_MONGO_CLIENT.get_collection('business', 'open_uri_log')
# 计算查询时间范围
now = datetime.datetime.now()
last = now - datetime.timedelta(hours=1)
now_year = now.year
now_month = now.month
now_day = now.day
now_hour = now.hour
last_year = last.year
last_month = last.month
last_day = last.day
last_hour = last.hour
def sync():
user_ids = []
df_mysql = pd.DataFrame(pd.read_sql("SELECT ID as user_id, IS_TEST FROM api_user WHERE IS_MONITOR = 1", con=engine))
for index in df_mysql.index:
user_id = df_mysql['user_id'].get(index)
user_ids.append(str(user_id))
df_mongo = pd.DataFrame(list(open_uri_log_conn.find(
{'user_id': {'$in': user_ids},
'c_date':
{'$gte': datetime.datetime(last_year, last_month, last_day, last_hour, 0),
'$lt': datetime.datetime(now_year, now_month, now_day, now_hour, 0)
}
}
).batch_size(10000)))
if len(df_mongo) > 0:
df_mongo = df_mongo.groupby(by=['user_id', 'user_name', 'request_uri', 'company_name']).agg(
{'_id': 'count', 'time': 'sum'}).reset_index()
df_mongo['user_id'] = df_mongo['user_id'].astype('int')
df = pd.merge(df_mongo, df_mysql, how='left', on=['user_id'])
df
text = ''
user_id_last = ''
for index in df.index:
user_id = df['user_id'].get(index)
request_uri = df['request_uri'].get(index)
count = df['_id'].get(index)
total_time = df['time'].get(index)
if user_id_last == user_id:
text = text + ' 使用的接口: ' + request_uri + ' 使用次数: ' + str(count) + ' 总计耗时: ' + str(total_time)
else:
user_name = df['user_name'].get(index)
company_name = df['company_name'].get(index)
if df['IS_TEST'].get(index) == 0:
user_type = '正式账户'
else:
user_type = '测试账户'
text = text + '\n' + company_name + ' 用户ID: ' + str(
user_id) + '(' + user_type + ')' + ' 用户名: ' + user_name + ' 使用的接口: ' + request_uri + ' 使用次数: ' + str(
count) + ' 总计耗时: ' + str(total_time)
user_id_last = user_id
sendDingDing(text)
def sendDingDing(text):
text = '【Api接口使用提醒】' + text
# robot_password = "SECf69d6dc5be4ca7e52b598d7eac75eb2e7d3bd6707c9f0a3ccb3b8a68ecf6f3fc"
# robot_webhook = "https://oapi.dingtalk.com/robot/send?access_token=f617e5b73808597d38101393b34c609cbe5353f9ed9a70cabda0b9e6e8ca371e"
token = '1e25f620ea2be926b2441e4f30d67f37abdff2e14088b03b41b4082b11fc34ea'
# 发送钉钉群消息
dingutils.send_text(text,token)
sync()
@@ -29,18 +29,27 @@ def do():
global count global count
is_old = False is_old = False
time_now = int(time.mktime(time.strptime(datetime.datetime.now().strftime('%Y-%m-%d'), '%Y-%m-%d'))) time_now = int(time.mktime(time.strptime(datetime.datetime.now().strftime('%Y-%m-%d'), '%Y-%m-%d')))
if 'open_date' in x and pd.notna(x['open_date']): try:
time_old = int(time.mktime(time.strptime(dateutils.format_date(dateutils.add_months(x['open_date'], years=3), '%Y-%m-%d'), '%Y-%m-%d'))) if 'open_date' in x and pd.notna(x['open_date']) and str(x['open_date']) != '':
if time_old < time_now: time_old = int(time.mktime(time.strptime(dateutils.format_date(dateutils.add_months(x['open_date'], years=3), '%Y-%m-%d'), '%Y-%m-%d')))
is_old = True if time_old < time_now:
if is_old is False and 'live_date' in x and pd.notna(x['live_date']): is_old = True
time_old = int(time.mktime(time.strptime(dateutils.format_date(dateutils.add_months(dateutils.parse_date(x['live_date']), years=3), '%Y-%m-%d'), '%Y-%m-%d'))) except Exception as e:
if time_old < time_now: log.error(e)
is_old = True try:
if is_old is False and 'create_year' in x and pd.notna(x['create_year']): if is_old is False and 'live_date' in x and pd.notna(x['live_date']) and str(x['live_date']) != '':
time_old = int(time.mktime(time.strptime(dateutils.format_date(dateutils.add_months(dateutils.parse_date(x['create_year'], '%Y-%m'), years=3), '%Y-%m-%d'), '%Y-%m-%d'))) time_old = int(time.mktime(time.strptime(dateutils.format_date(dateutils.add_months(dateutils.parse_date(x['live_date']), years=3), '%Y-%m-%d'), '%Y-%m-%d')))
if time_old < time_now: if time_old < time_now:
is_old = True is_old = True
except Exception as e:
log.error(e)
try:
if is_old is False and 'create_year' in x and pd.notna(x['create_year']) and str(x['create_year']) != '':
time_old = int(time.mktime(time.strptime(dateutils.format_date(dateutils.add_months(dateutils.parse_date(x['create_year'], '%Y'), years=3), '%Y-%m-%d'), '%Y-%m-%d')))
if time_old < time_now:
is_old = True
except Exception as e:
log.error(e)
# 查询样本库 # 查询样本库
names = [x['name']] names = [x['name']]
if type(x['alias']) == list: if type(x['alias']) == list:
+88 -79
View File
@@ -3,46 +3,71 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils, areautils from goojia_utils import ElasticSearchClient, commutils, dateutils, areautils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
import numpy as np
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
import sys import sys
from threading import Lock
mutex = Lock()
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.206')
es89_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
m_date = dateutils.iso_format() m_date = dateutils.iso_format()
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
file_path = "/opt/airflow/dags/repo/dags/business/business/community/data"
avg_cost_min_max_df = pd.read_excel(file_path + '/全国各区域人均消费中位值.xlsx')
avg_cost_city_region_dict = dict()
def find_around_price(lng, lat): # 加载各区域人均消费最大最小值
def deal_avg_cost(x):
city_now = x['city']
region_now = x['region']
if pd.notna(city_now) and pd.notna(region_now):
avg_cost_city_region_dict[city_now + region_now] = {'min': x['3'], 'max': x['97']}
elif pd.notna(city_now):
avg_cost_city_region_dict[city_now] = {'min': x['3'], 'max': x['97']}
avg_cost_min_max_df.apply(lambda x: deal_avg_cost(x), axis=1)
def find_around_price(seaweed):
""" """
查找周边均价 查找周边均价
返回取值为10~300,为所有有价格的1-99百分位
""" """
if pd.isna(lng): lng = commutils.format_lat_lng(seaweed['lng'])
return None lat = commutils.format_lat_lng(seaweed['lat'])
city_short = seaweed['city']
region_short = seaweed['region']
city_display = areautils.get_city_display(city_short)
filter_query = [{"terms": {"tag": ["美食;中餐厅", "美食;外国餐厅"]}}]
if pd.notna(city_display):
filter_query.append({"term": {"city": {"value": city_display}}})
min_now = 10
max_now = 300
if (city_short+region_short) in avg_cost_city_region_dict:
min_now = avg_cost_city_region_dict[city_short + region_short]['min']
max_now = avg_cost_city_region_dict[city_short + region_short]['max']
elif city_short in avg_cost_city_region_dict:
min_now = avg_cost_city_region_dict[city_short]['min']
max_now = avg_cost_city_region_dict[city_short]['max']
filter_query.append({"range": {"price": {"gte": min_now, "lte": max_now}}})
filter_query.append({"geo_distance": {"distance": 1000, "location": [lng, lat]}})
ret_es = es206_client.elastic_client.search({ ret_es = es206_client.elastic_client.search({
"query": { "query": {
"bool": { "bool": {
"must": [ "filter": filter_query
{
"term": {
"status": {
"value": 0
}
}
}
],
"filter": [
{
"geo_distance": {
"distance": 1000,
"location": [lng, lat]
}
}
]
} }
}, },
"size": 0, "size": 0,
@@ -59,72 +84,51 @@ def find_around_price(lng, lat):
def build_doc(item): def build_doc(item):
doc_dict = dict() doc_dict = dict()
if 'avg_cost' in item and pd.notna(item['avg_cost']): if 'avg_cost' in item:
doc_dict['avg_cost'] = item['avg_cost'] if pd.notna(item['avg_cost']) or item['avg_cost'] > 0:
doc_dict['avg_cost'] = int(item['avg_cost'])
else:
doc_dict['avg_cost'] = None
return doc_dict return doc_dict
def update_status(new_df): find_count = 0
actions = list()
def deal(item):
action_doc = {
"_op_type": "update",
"_index": "bd_poi",
"_id": item['_id'],
"doc": {'status': 0}
}
actions.append(action_doc)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
actions.clear()
return item
new_df.apply(lambda x: deal(x), axis=1) def find_around_price_item(seaweed):
if len(actions) > 0: global find_count
helpers.bulk(es206_client.elastic_client, actions) find_count += 1
actions.clear() if find_count % 1000 == 0:
log.info("已查找" + str(find_count))
if 'lng' not in seaweed:
return
price = find_around_price(seaweed)
mutex.acquire(timeout=60)
if pd.notna(price):
seaweed['avg_cost'] = int(price)
else:
seaweed['avg_cost'] = None
mutex.release()
def sync(city=None): def sync(city=None):
log.info("开始同步消费数据...") log.info("开始同步消费数据...")
query_sql = "select id, city, tag, type, price from bd_poi where price > 0 and tag in('美食;中餐厅', '美食;外国餐厅') " query = {}
if pd.notna(city): if pd.notna(city):
query_sql += " and city='" + areautils.get_city_display(city) + "'" query['city'] = city
df = es206_client.scan_by_sql_to_data_frame(query_sql) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
# 按百分位5-95筛选数据 seaweed_list = list()
cities = df['city'].unique() seaweed_df.apply(lambda x: seaweed_list.append(dict(x)), axis=1)
for city_now in cities:
df.loc[df['city'] == city_now, 'minPrice5'] = np.percentile(a=df[df['city'] == city_now]['price'], q=5, axis=0)
df.loc[df['city'] == city_now, 'minPrice95'] = np.percentile(a=df[df['city'] == city_now]['price'], q=95, axis=0)
new_df = df[(df['price'] >= df['minPrice5']) & (df['price'] <= df['minPrice95'])]
new_df['new_mean'] = new_df.groupby(by=['city'])['price'].transform(np.mean)
# 按最新的百分位更新库中价格状态
update_status(new_df)
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8')
if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy()
seaweed_df = seaweed_df.reset_index(drop=True)
def find_around_price_item(index, item):
if index % 1000 == 0:
log.info("已查找" + str(index))
if 'lng' not in item:
return
lng = commutils.format_lat_lng(item['lng'])
lat = commutils.format_lat_lng(item['lat'])
price = find_around_price(lng, lat)
seaweed_df.loc[index, 'avg_cost'] = price
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
for index, item in seaweed_df.iterrows(): for seaweed in seaweed_list:
pool.submit(find_around_price_item, index, item) pool.submit(find_around_price_item, seaweed)
# for index, item in seaweed_df.iterrows():
# find_around_price_item(index, item) # for seaweed in seaweed_list:
# find_around_price_item(seaweed)
seaweed_df = pd.DataFrame(seaweed_list)
actions = list() actions = list()
count = 0 count = 0
for index, item in seaweed_df.iterrows(): for index, item in seaweed_df.iterrows():
@@ -140,16 +144,21 @@ def sync(city=None):
actions.append(action) actions.append(action)
if len(actions) == 5000: if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions) helpers.bulk(es89_client.elastic_client, actions)
count += len(actions) count += len(actions)
log.info("更新:" + str(count)) log.info("更新:" + str(count))
actions.clear() actions.clear()
if len(actions) > 0: if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions) helpers.bulk(es89_client.elastic_client, actions)
actions.clear() actions.clear()
log.info("完成同步消费数据...") log.info("完成同步消费数据...")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -3,14 +3,15 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils from goojia_utils import ElasticSearchClient, commutils, dateutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock from threading import Lock
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -89,10 +90,10 @@ def deal(actions, item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步商圈至206 {index_name}") log.info(f"开始同步商圈至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
actions = list() actions = list()
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
@@ -105,5 +106,10 @@ def sync(city=None):
log.info(f"完成同步商圈至206 {index_name}") log.info(f"完成同步商圈至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -0,0 +1,121 @@
"""
城市等级及城市价格等级计算
"""
from goojia_utils import ElasticSearchClient
import pandas as pd
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import QuantileTransformer
import logging
from elasticsearch import helpers
import sys
def sync():
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
log = logging.getLogger(__name__)
path = "/opt/airflow/dags/repo/dags/business/business/community/data"
new_df = pd.read_excel(path + '/城市等级及城市价格等级信息.xlsx')
community_summary_df = es206_client.scan_by_sql_to_data_frame(
'select city, region, name, consumption_potential from community_summary')
merge_df = pd.merge(community_summary_df, new_df, how='left', on=['city'])
city_grade = ['一线', '新一线', '二线', '三四线', '五线']
rating = [1, 2, 3, 4, 5]
index_name = "community_summary"
def filter_min_max(community_summary_df, source):
community_summary_df['normal_consumption_potential'] = QuantileTransformer(
output_distribution="normal").fit_transform(community_summary_df[[source]])
return community_summary_df
def gmm_train(total_rate, consumption_df, source, target):
# 高斯混合模型
gmm = GaussianMixture(n_components=total_rate, covariance_type='tied', max_iter=200, n_init=10).fit(
consumption_df[[source]])
consumption_df.loc[:, 'cluster'] = gmm.predict(consumption_df[[source]])
consumption_df.loc[:, 'price_mean_cluster'] = consumption_df.groupby('cluster')[source].transform('mean')
consumption_df.loc[:, target] = consumption_df['price_mean_cluster'].rank(method='dense',
ascending=False).astype(
int)
consumption_df = consumption_df.drop(columns=['cluster', 'price_mean_cluster'], axis=1)
return consumption_df
log.info("开始按城市等级评级...")
merge_df = merge_df[merge_df['consumption_potential'].notna()].copy()
merge_df['consumption_potential_city_grade'] = merge_df['consumption_potential'] * merge_df['city_grade_ratio']
merge_df['consumption_potential_city_grade'].fillna(merge_df['consumption_potential'], inplace=True)
city_grade_ret_df = pd.DataFrame()
for grade in city_grade:
temp_df = merge_df[merge_df['city_grade'] == grade].copy()
log.info(f"当前城市等级{grade}, 评级数量{len(temp_df)}")
temp_df = filter_min_max(temp_df, 'consumption_potential_city_grade')
source = "normal_consumption_potential"
target = "city_level_rating"
temp_df = gmm_train(5, temp_df, source, target)
city_grade_ret_df = pd.concat([city_grade_ret_df, temp_df])
log.info("完成按城市等级评级...")
log.info("开始按城市价格等级评级")
merge_df = merge_df[merge_df['consumption_potential'].notna()].copy()
merge_df['consumption_potential_rating'] = merge_df['consumption_potential'] * merge_df['rating_ratio']
merge_df['consumption_potential_rating'].fillna(merge_df['consumption_potential'], inplace=True)
rating_ret_df = pd.DataFrame()
for grade in rating:
temp_df = merge_df[merge_df['rating'] == grade].copy()
log.info(f"当前城市价格等级{grade}, 评级数量{len(temp_df)}")
temp_df = filter_min_max(temp_df, 'consumption_potential_rating')
source = "normal_consumption_potential"
target = "city_price_level_rating"
temp_df = gmm_train(5, temp_df, source, target)
rating_ret_df = pd.concat([rating_ret_df, temp_df])
log.info("完成按城市价格等级评级")
city_grade_ret_df = city_grade_ret_df[['_id', 'city', 'region', 'name', 'city_grade', 'city_level_rating']]
rating_ret_df = rating_ret_df[['_id', 'city', 'region', 'name', 'rating', 'city_price_level_rating']]
aim_df = pd.merge(city_grade_ret_df, rating_ret_df, how='outer', on=['_id', 'city', 'region', 'name'])
log.info(f"开始写入ES206索引{index_name}")
actions = list()
count = 0
city_grade_dict = {'一线': 1, '新一线': 2, '二线': 3, '三四线': 4, '五线': 5}
for index, item in aim_df.iterrows():
row = dict()
_id = str(item["_id"])
if pd.notna(item['city_grade']):
row['city_level'] = city_grade_dict[item['city_grade']]
if pd.notna(item['city_level_rating']):
row['city_level_rating'] = item['city_level_rating']
if pd.notna(item['rating']):
row['city_price_level'] = int(item['rating'])
if pd.notna(item['city_price_level_rating']):
row['city_price_level_rating'] = item['city_price_level_rating']
action = {'_op_type': 'update',
'_index': index_name,
'_id': _id, # 请勿忘记ID
'doc': row,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("评级中: 当前已评级{}".format(count))
if len(actions) > 0:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("评级完成,共更新: {}".format(count))
log.info(f"完成写入ES206索引{index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None
# 对于该评级等级来说,不能单个城市处理,只能全量
if pd.isna(city):
sync()
+108 -9
View File
@@ -1,9 +1,39 @@
""" """
同步小区基本信息至ES206 同步小区基本信息至ES206
""" """
import os
import time
# 强制设置 Python 进程内部本地时区为北京时间(CST / UTC+8),消除 Docker 容器内的 UTC 时区日志差异
os.environ['TZ'] = 'Asia/Shanghai'
if hasattr(time, 'tzset'):
time.tzset()
import logging import logging
import pandas as pd import pandas as pd
import numpy as np
# 兼容性补丁:解决 goojia_utils.ClickHouseClient.to_sql_plus 在将 DataFrame 写入 ClickHouse 时,
# 对含有 list/array 类型的列(如 alias)进行 pd.notna(item) 判定时导致 ValueError 的 bug
_orig_notna = pd.notna
def patched_notna(obj):
if isinstance(obj, (list, tuple, np.ndarray, dict, set)):
return True
try:
res = _orig_notna(obj)
if hasattr(res, 'any'):
return bool(res.any())
elif isinstance(res, (list, np.ndarray)):
return any(res)
return res
except Exception:
return True
pd.notna = patched_notna
pd.isna = lambda x: not patched_notna(x)
import pandas
pandas.notna = patched_notna
pandas.isna = lambda x: not patched_notna(x)
from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, dateutils from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, dateutils
from elasticsearch import helpers from elasticsearch import helpers
import os import os
@@ -11,7 +41,8 @@ import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed') seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
m_date = dateutils.iso_format() m_date = dateutils.iso_format()
@@ -47,7 +78,13 @@ def build_doc(item):
estate_charge = commutils.format_property_fee(item['estate_charge']) estate_charge = commutils.format_property_fee(item['estate_charge'])
if pd.notna(estate_charge): if pd.notna(estate_charge):
doc_dict['estate_charge'] = float(estate_charge) # 剔除过大物业费导致ES报错
if float(estate_charge) <= 1000:
doc_dict['estate_charge'] = float(estate_charge)
# 表示该物业费已经从库中补充
doc_dict['estate_charge_flag'] = 0
# 先打10分,对于其他的流程在修改分数
doc_dict['credit_detail'] = {'estate_charge_score': 10}
finish_building_date = commutils.format_finish_building_date(item['finish_building_date']) finish_building_date = commutils.format_finish_building_date(item['finish_building_date'])
if pd.notna(finish_building_date): if pd.notna(finish_building_date):
@@ -57,6 +94,31 @@ def build_doc(item):
lat = commutils.format_lat_lng(item['lat']) lat = commutils.format_lat_lng(item['lat'])
if pd.notna(lng) and pd.notna(lat): if pd.notna(lng) and pd.notna(lat):
doc_dict['location'] = {'lon': lng, 'lat': lat} doc_dict['location'] = {'lon': lng, 'lat': lat}
# 新增字段存储
all_area = commutils.format_all_area(item['all_area'])
if pd.notna(all_area):
doc_dict['all_area'] = int(all_area)
build_area = commutils.format_build_area(item['build_area'])
if pd.notna(build_area):
doc_dict['build_area'] = int(build_area)
greening_rate = commutils.format_greening_rate(item['virescence_percent'])
if pd.notna(greening_rate):
doc_dict['greening_rate'] = greening_rate
volume_rate = commutils.format_plot_ratio(item['cubage_percent'])
if pd.notna(volume_rate):
doc_dict['volume_rate'] = volume_rate
house_type_area_dic = commutils.format_house_type_area(item['main_type'], item['main_type_area'])
main_house_type = house_type_area_dic['house_type']
main_house_area = house_type_area_dic['house_area']
if pd.notna(main_house_area):
try:
doc_dict['main_house_area'] = int(float(main_house_area))
except Exception as e:
pass
if pd.notna(main_house_type):
doc_dict['main_house_type'] = main_house_type
return doc_dict return doc_dict
@@ -84,21 +146,35 @@ def deal_comm(community_summary_df, seaweed_df):
def sync(city=None): def sync(city=None):
log.info(f"开始同步小区基本属性至206 {index_name}...") log.info(f"开始同步小区基本属性至206 {index_name}...")
seaweed_query = {'status': 0, 'cat': 'district'} seaweed_query = {'status': 0, 'cat': 'district'}
mongo_delete_query = {}
if pd.notna(city): if pd.notna(city):
seaweed_query['city'] = city seaweed_query['city'] = city
mongo_delete_query['city'] = city
community_summary_conn.delete_many(mongo_delete_query)
seaweed_df = pd.DataFrame( seaweed_df = pd.DataFrame(
list(seaweed_conn.find(seaweed_query, {'city': 1, 'region': 1, 'name': 1, 'lng': 1, 'lat': 1, list(seaweed_conn.find(seaweed_query, {'city': 1, 'region': 1, 'name': 1, 'lng': 1, 'lat': 1,
'alias': 1, 'estate_type2': 1, 'build_type': 1, 'alias': 1, 'estate_type2': 1, 'build_type': 1,
'household_count': 1, 'block': 1, 'household_count': 1, 'block': 1,
'estate_charge': 1, 'finish_building_date': 1}) 'estate_charge': 1, 'finish_building_date': 1,
'all_area': 1, 'build_area': 1, 'main_type': 1, 'main_type_area': 1, 'cubage_percent': 1, 'virescence_percent': 1
})
.batch_size(100000))) .batch_size(100000)))
sql = f"select id from {index_name} " sql = f"select id from {index_name} "
if pd.notna(city): if pd.notna(city):
sql += f" where city='{city}'" sql += f" where city='{city}'"
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql) community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
seaweed_df['_id'] = seaweed_df['_id'].astype(str) seaweed_df['_id'] = seaweed_df['_id'].astype(str)
# 清除别名中逗号
def dealAlias(x):
if isinstance(x, list) and len(x) > 0:
for index, element in enumerate(x):
if "'" in element:
x[index] = str(element).replace("'", "")
return x
seaweed_df['alias'] = seaweed_df['alias'].apply(lambda x: dealAlias(x))
if len(community_summary_df) > 0: if len(community_summary_df) > 0:
community_summary_df['flag'] = 1 community_summary_df['flag'] = 1
@@ -109,6 +185,19 @@ def sync(city=None):
seaweed_existed_df = pd.DataFrame() seaweed_existed_df = pd.DataFrame()
seaweed_not_existed_df = seaweed_df seaweed_not_existed_df = seaweed_df
# 写入csv文件,防止小区变动导致的后续过程更新报错,
# seaweed_df.to_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8', index=False)
# 写入中间集合 MongoDB, 解决分布式 Worker 节点的共享问题
insert_cols = ['_id', 'city', 'region', 'name', 'lng', 'lat', 'alias', 'estate_type2', 'build_type',
'household_count', 'block', 'estate_charge', 'finish_building_date']
records = seaweed_df[insert_cols].to_dict(orient='records')
if len(records) > 0:
log.info(f"正在写入 MongoDB community_summary 中间集合,共 {len(records)} 条记录...")
batch_size = 5000
for i in range(0, len(records), batch_size):
community_summary_conn.insert_many(records[i:i + batch_size])
log.info("MongoDB community_summary 写入完毕。")
actions = list() actions = list()
# 对于存在的更新 # 对于存在的更新
count = 0 count = 0
@@ -128,6 +217,11 @@ def sync(city=None):
count += len(actions) count += len(actions)
log.info("更新:" + str(count)) log.info("更新:" + str(count))
actions.clear() actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新:" + str(count))
actions.clear()
# 对于不存在的插入 # 对于不存在的插入
count = 0 count = 0
@@ -138,7 +232,8 @@ def sync(city=None):
"_op_type": "index", "_op_type": "index",
"_index": index_name, "_index": index_name,
"_id": item['_id'], "_id": item['_id'],
"_source": doc_info "_source": doc_info,
"_retry_on_conflict": 5
} }
actions.append(action) actions.append(action)
@@ -149,14 +244,18 @@ def sync(city=None):
actions.clear() actions.clear()
if len(actions) > 0: if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions) helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("插入:" + str(count))
actions.clear() actions.clear()
# 写入csv文件,防止小区变动导致的后续过程更新报错
seaweed_df.to_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8', index=False)
deal_comm(community_summary_df, seaweed_df) deal_comm(community_summary_df, seaweed_df)
log.info(f"完成同步小区基本属性至206 {index_name}") log.info(f"完成同步小区基本属性至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+245 -67
View File
@@ -1,6 +1,8 @@
""" """
月消费潜力、评级计算 月消费潜力、评级计算
""" """
import warnings
from goojia_utils import ElasticSearchClient, dateutils, nextcloudutils from goojia_utils import ElasticSearchClient, dateutils, nextcloudutils
import pandas as pd import pandas as pd
import numpy as np import numpy as np
@@ -9,13 +11,16 @@ import os
from elasticsearch import helpers from elasticsearch import helpers
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import sys import sys
from scipy.stats import boxcox
from sklearn.preprocessing import RobustScaler
from sklearn.mixture import GaussianMixture from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import QuantileTransformer from sklearn.preprocessing import QuantileTransformer
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
# 忽略警告
es206_client = ElasticSearchClient(host='192.168.0.206') warnings.filterwarnings("ignore", category=UserWarning)
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
path = "/opt/airflow/dags/repo/dags/business/business/community/data" path = "/opt/airflow/dags/repo/dags/business/business/community/data"
save_path_prefix = "/home/airflow/data/community" save_path_prefix = "/home/airflow/data/community"
@@ -28,18 +33,41 @@ nextcloud_aim_file_path = '共享/财力评级/财力评级图'
reasonable_df = pd.read_excel(path + '/财力评级分布情况20221215.xlsx') reasonable_df = pd.read_excel(path + '/财力评级分布情况20221215.xlsx')
reasonable_dict = reasonable_df.set_index('city')['是否合理'].to_dict() reasonable_dict = reasonable_df.set_index('city')['是否合理'].to_dict()
scaler = RobustScaler()
# Min-Max 归一化到 [1, 100]
def min_max_scale(df, column, target_column, new_min=1, new_max=100):
X_min = df[column].min()
X_max = df[column].max()
# 如果存在缺失值,则不进行归一化
if pd.isna(X_min):
return df
# 防止分母为零(如果所有值相等)
if X_max == X_min:
df[target_column] = new_min # 如果所有值相同,设为下界
else:
df[target_column] = (new_min + (df[column] - X_min) * (new_max - new_min) / (X_max - X_min))
df[target_column] = df[target_column].astype(int)
return df
def calc_consumption_potential(city): def calc_consumption_potential(city):
""" """
计算消费潜力 计算消费潜力
""" """
sql = "select id, city, region, name, total_price, estate_charge, relative_price_ranking, avg_cost from community_summary where city='" + city + "'" sql = "select id, city, region, name, total_price, estate_charge, estate_charge_fix, relative_price_ranking, avg_cost from community_summary where city='" + city + "'"
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql) community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
columns = community_summary_df.columns columns = community_summary_df.columns
if 'total_price' not in columns or 'estate_charge' not in columns or 'relative_price_ranking' not in columns or 'avg_cost' not in columns: if 'total_price' not in columns or 'estate_charge' not in columns or 'relative_price_ranking' not in columns or 'avg_cost' not in columns:
return [] return []
community_summary_df = community_summary_df.dropna(how='any').copy() # 解决某些城市无修正物业费问题
if 'estate_charge_fix' not in community_summary_df.columns:
community_summary_df['estate_charge_fix'] = None
# 修正的物业费入模
community_summary_df['estate_charge_fix'].fillna(community_summary_df['estate_charge'], inplace=True)
#community_summary_df = community_summary_df.dropna(how='any').copy()
if len(community_summary_df) == 0: if len(community_summary_df) == 0:
log.warning(f"{city}无法评级,缺少因子") log.warning(f"{city}无法评级,缺少因子")
return [] return []
@@ -47,13 +75,47 @@ def calc_consumption_potential(city):
# 户均总价计算时候改成万 # 户均总价计算时候改成万
community_summary_df['total_price'] = round(community_summary_df['total_price'] / 10000, 0) community_summary_df['total_price'] = round(community_summary_df['total_price'] / 10000, 0)
community_summary_df['total_price_mean'] = community_summary_df.groupby(by=['city'])['total_price'].transform( community_summary_df['total_price_fixed'] = community_summary_df['total_price']
np.mean) community_summary_df['avg_cost_fixed'] = community_summary_df['avg_cost']
community_summary_df['total_price_std'] = community_summary_df.groupby(by=['city'])['total_price'].transform(np.std)
community_summary_df['estate_charge_mean'] = community_summary_df.groupby(by=['city'])['estate_charge'].transform( community_summary_df['estate_charge_fixed'] = community_summary_df['estate_charge_fix']
# community_summary_df['total_price_fixed'] = QuantileTransformer(output_distribution="normal").fit_transform(
# community_summary_df[['total_price']])
# community_summary_df['avg_cost_fixed'] = QuantileTransformer(output_distribution="normal").fit_transform(
# community_summary_df[['avg_cost']])
#
# community_summary_df['estate_charge_fixed'] = QuantileTransformer(output_distribution="normal").fit_transform(
# community_summary_df[['estate_charge_fix']])
# 异常值处理
total_price_fixed_threshold = community_summary_df['total_price_fixed'].quantile(0.99)
community_summary_df['total_price_fixed'] = np.where(community_summary_df['total_price_fixed'] > total_price_fixed_threshold, total_price_fixed_threshold, community_summary_df['total_price_fixed'])
max_total_price_fixed = community_summary_df['total_price_fixed'].max()
community_summary_df['total_price_fixed'] = np.log(community_summary_df['total_price_fixed'] + 1) / np.log(max_total_price_fixed + 1)
# 人均消费极大值处理
avg_cost_fixed_threshold = community_summary_df['avg_cost_fixed'].quantile(0.99)
community_summary_df['avg_cost_fixed'] = np.where(community_summary_df['avg_cost_fixed'] > avg_cost_fixed_threshold, avg_cost_fixed_threshold, community_summary_df['avg_cost_fixed'])
max_avg_cost_fixed = community_summary_df['avg_cost_fixed'].max()
try:
#community_summary_df['avg_cost_fixed'] = np.log1p(community_summary_df['avg_cost_fixed'])
community_summary_df['avg_cost_fixed'] = np.log(community_summary_df['avg_cost_fixed'] + 1) / np.log(max_avg_cost_fixed + 1)
except Exception:
community_summary_df['avg_cost_fixed'] = None
pass
#community_summary_df['avg_cost_fixed'] = 1 - np.log(community_summary_df['avg_cost_fixed'] + 1) / np.log(max_avg_cost_fixed + 1)
estate_charge_fixed_threshold = community_summary_df['estate_charge_fixed'].quantile(0.99)
community_summary_df['estate_charge_fixed'] = np.where(community_summary_df['estate_charge_fixed'] > estate_charge_fixed_threshold, estate_charge_fixed_threshold, community_summary_df['estate_charge_fixed'])
max_estate_charge_fixed = community_summary_df['estate_charge_fixed'].max()
community_summary_df['estate_charge_fixed'] = np.log(community_summary_df['estate_charge_fixed'] + 1) / np.log(max_estate_charge_fixed + 1)
#relative_price_ranking_threshold = community_summary_df['relative_price_ranking'].quantile(0.99)
#community_summary_df['relative_price_ranking'] = np.where(community_summary_df['relative_price_ranking'] > relative_price_ranking_threshold, relative_price_ranking_threshold, community_summary_df['relative_price_ranking'])
max_relative_price_ranking_fixed = community_summary_df['relative_price_ranking'].max()
community_summary_df['relative_price_ranking'] = np.log(community_summary_df['relative_price_ranking'] + 1) / np.log(max_relative_price_ranking_fixed + 1)
community_summary_df['estate_charge_mean'] = community_summary_df.groupby(by=['city'])['estate_charge_fixed'].transform(
np.mean) np.mean)
community_summary_df['estate_charge_std'] = community_summary_df.groupby(by=['city'])['estate_charge'].transform( community_summary_df['estate_charge_std'] = community_summary_df.groupby(by=['city'])['estate_charge_fixed'].transform(
np.std) np.std)
community_summary_df['relative_price_ranking_mean'] = community_summary_df.groupby(by=['city'])[ community_summary_df['relative_price_ranking_mean'] = community_summary_df.groupby(by=['city'])[
@@ -61,17 +123,25 @@ def calc_consumption_potential(city):
community_summary_df['relative_price_ranking_std'] = community_summary_df.groupby(by=['city'])[ community_summary_df['relative_price_ranking_std'] = community_summary_df.groupby(by=['city'])[
'relative_price_ranking'].transform(np.std) 'relative_price_ranking'].transform(np.std)
community_summary_df['avg_cost_mean'] = community_summary_df.groupby(by=['city'])['avg_cost'].transform(np.mean) try:
community_summary_df['avg_cost_std'] = community_summary_df.groupby(by=['city'])['avg_cost'].transform(np.std) community_summary_df['avg_cost_mean'] = community_summary_df.groupby(by=['city'])['avg_cost_fixed'].transform(np.mean)
community_summary_df['avg_cost_std'] = community_summary_df.groupby(by=['city'])['avg_cost_fixed'].transform(np.std)
except Exception:
community_summary_df['avg_cost_mean'] = None
community_summary_df['avg_cost_std'] = None
community_summary_df['total_price_mean'] = community_summary_df.groupby(by=['city'])['total_price_fixed'].transform(
np.mean)
community_summary_df['total_price_std'] = community_summary_df.groupby(by=['city'])['total_price_fixed'].transform(np.std)
# 和田等小城市均值可能为0 # 和田等小城市均值可能为0
if community_summary_df['total_price_std'].unique()[0] != 0: if community_summary_df['total_price_std'].unique()[0] != 0:
community_summary_df['total_price_score'] = 286.0134196 * (community_summary_df['total_price'] - community_summary_df['total_price_mean']) / community_summary_df['total_price_std'] community_summary_df['total_price_score'] = 286.0134196 * (community_summary_df['total_price_fixed'] - community_summary_df['total_price_mean']) / community_summary_df['total_price_std']
else: else:
community_summary_df['total_price_score'] = 0 community_summary_df['total_price_score'] = 0
if community_summary_df['estate_charge_std'].unique()[0] != 0: if community_summary_df['estate_charge_std'].unique()[0] != 0:
community_summary_df['estate_charge_score'] = 139.1374524 * (community_summary_df['estate_charge'] - community_summary_df['estate_charge_mean']) / community_summary_df['estate_charge_std'] community_summary_df['estate_charge_score'] = 139.1374524 * (community_summary_df['estate_charge_fixed'] - community_summary_df['estate_charge_mean']) / community_summary_df['estate_charge_std']
else: else:
community_summary_df['estate_charge_score'] = 0 community_summary_df['estate_charge_score'] = 0
@@ -81,18 +151,26 @@ def calc_consumption_potential(city):
community_summary_df['relative_price_ranking_score'] = 0 community_summary_df['relative_price_ranking_score'] = 0
if community_summary_df['avg_cost_std'].unique()[0] != 0: if community_summary_df['avg_cost_std'].unique()[0] != 0:
community_summary_df['avg_cost_score'] = 67.0195871700 * (community_summary_df['avg_cost'] - community_summary_df['avg_cost_mean']) / community_summary_df['avg_cost_std'] community_summary_df['avg_cost_score'] = 67.0195871700 * (community_summary_df['avg_cost_fixed'] - community_summary_df['avg_cost_mean']) / community_summary_df['avg_cost_std']
else: else:
community_summary_df['avg_cost_score'] = 0 community_summary_df['avg_cost_score'] = 0
community_summary_df['consumption_potential'] = community_summary_df['total_price_score'] + community_summary_df[ community_summary_df['consumption_potential'] = community_summary_df['total_price_score'] + community_summary_df[
'estate_charge_score'] + community_summary_df['relative_price_ranking_score'] + community_summary_df['avg_cost_score'] + community_summary_df['avg_cost_score'] + 4099.978248 'estate_charge_score'] + community_summary_df['relative_price_ranking_score'] + community_summary_df['avg_cost_score'] + 4099.978248
return community_summary_df return community_summary_df
def gmm_train(total_rate, consumption_df, source, target): def gmm_train(total_rate, consumption_df, source, target):
# 高斯混合模型 # 高斯混合模型
gmm = GaussianMixture(n_components=total_rate, covariance_type='tied', max_iter=200, n_init=10).fit(consumption_df[[source]]) gmm = GaussianMixture(
n_components=total_rate,
covariance_type='full', # 更灵活的协方差矩阵
max_iter=200,
n_init=10,
reg_covar=1e-6, # 正则化防止奇异矩阵
random_state=42
).fit(consumption_df[[source]])
consumption_df = consumption_df.copy()
consumption_df.loc[:, 'cluster'] = gmm.predict(consumption_df[[source]]) consumption_df.loc[:, 'cluster'] = gmm.predict(consumption_df[[source]])
consumption_df.loc[:, 'price_mean_cluster'] = consumption_df.groupby('cluster')[source].transform('mean') consumption_df.loc[:, 'price_mean_cluster'] = consumption_df.groupby('cluster')[source].transform('mean')
consumption_df.loc[:, target] = consumption_df['price_mean_cluster'].rank(method='dense', ascending=False).astype(int) consumption_df.loc[:, target] = consumption_df['price_mean_cluster'].rank(method='dense', ascending=False).astype(int)
@@ -104,27 +182,86 @@ def calc_consumption_rating(community_summary_df):
""" """
计算消费等级 计算消费等级
""" """
df = community_summary_df[['_id', 'city', 'consumption_potential']] df = community_summary_df[['_id', 'city', 'consumption_potential', 'total_price_score', 'estate_charge_score', 'relative_price_ranking_score', 'avg_cost_score']]
df = df.dropna(subset=['consumption_potential']) #df = df.dropna(subset=['consumption_potential'])
source = "normal_consumption_potential" source = "normal_consumption_potential"
target = "consumption_rating" target = "consumption_rating"
actions = list() actions = list()
count = 0 count = 0
if len(df) < 5: # 总评级
total_rate = 6
score_df = pd.DataFrame()
if len(df.dropna(subset=['consumption_potential'])) < total_rate:
# 对于样本低于5的直接排名 # 对于样本低于5的直接排名
df = df.dropna(subset=['consumption_potential'])
df[target] = df['consumption_potential'].rank(method='dense', ascending=False).astype(int) df[target] = df['consumption_potential'].rank(method='dense', ascending=False).astype(int)
df["total_price_rating"] = df['total_price_score'].rank(method='dense', ascending=False).astype(int)
df["estate_charge_rating"] = df['estate_charge_score'].rank(method='dense', ascending=False).astype(int)
df["relative_price_rating"] = df['relative_price_ranking_score'].rank(method='dense', ascending=False).astype(int)
df["avg_cost_rating"] = df['avg_cost_score'].rank(method='dense', ascending=False).astype(int)
else: else:
# 极值设置 # 极值设置
df = filter_min_max(df) score_df = gmm_train(total_rate, df.dropna(subset=['total_price_score']), "total_price_score", "total_price_rating")
df = gmm_train(5, df, source, target) score_df['robust_total_price_score'] = scaler.fit_transform(score_df[['total_price_score']])
# 批量更新 score_df = min_max_scale(score_df, 'robust_total_price_score', 'total_price_score_minmax')
df = df.copy()
df_threshold = community_summary_df['consumption_potential'].quantile(0.99)
community_summary_df['consumption_potential'] = np.where(community_summary_df['consumption_potential'] > df_threshold, df_threshold, community_summary_df['consumption_potential'])
df.loc[:, 'normal_consumption_potential'] = QuantileTransformer(output_distribution="normal").fit_transform(df[['consumption_potential']])
if len(df.dropna(subset=['consumption_potential'])) >= total_rate:
df = gmm_train(total_rate, df.dropna(subset=['consumption_potential']), source, target)
# 新增子因子评级
df = gmm_train(total_rate, df, "estate_charge_score", "estate_charge_rating")
df = gmm_train(total_rate, df, "relative_price_ranking_score", "relative_price_rating")
df = gmm_train(total_rate, df, "avg_cost_score", "avg_cost_rating")
# 新增子因子评分
df = min_max_scale(df, 'consumption_potential', 'consumption_score')
df = min_max_scale(df, 'estate_charge_score', 'estate_charge_score_minmax')
df = min_max_scale(df, 'relative_price_ranking_score', 'relative_price_ranking_score_minmax')
df = min_max_scale(df, 'avg_cost_score', 'avg_cost_score_minmax')
if len(score_df) > 0:
# 补充财力评级
df = pd.merge(score_df[['_id', 'total_price_rating', 'total_price_score_minmax']], df, how='left', on=['_id'])
# 填充缺失值
if 'consumption_rating' not in df.columns:
df["consumption_rating"] = None
if 'consumption_score' not in df.columns:
df["consumption_score"] = None
df["consumption_rating"] = df["consumption_rating"].fillna(df["total_price_rating"])
df["consumption_score"] = df["consumption_score"].fillna(df["total_price_score_minmax"])
# 批量更新 如果财力评级与户均总价评级都没有,不更新
for index, item in df.iterrows(): for index, item in df.iterrows():
row = dict() row = dict()
_id = str(item["_id"]) _id = str(item["_id"])
row['consumption_rating'] = item['consumption_rating'] if 'consumption_rating' in item and pd.notna(item['consumption_rating']):
row['consumption_potential'] = item['consumption_potential'] row['consumption_rating'] = item['consumption_rating']
if 'consumption_score' in item and pd.notna(item['consumption_score']):
row['consumption_score'] = item['consumption_score']
if 'consumption_potential' in item and pd.notna(item['consumption_potential']):
row['consumption_potential'] = item['consumption_potential']
row['m_date'] = dateutils.iso_format() row['m_date'] = dateutils.iso_format()
row['sub_factor'] = {}
if 'total_price_rating' in item and pd.notna(item['total_price_rating']):
row['sub_factor']['total_price_rating'] = item['total_price_rating']
if 'estate_charge_rating' in item and pd.notna(item['estate_charge_rating']):
row['sub_factor']['estate_charge_rating'] = item['estate_charge_rating']
if 'relative_price_rating' in item and pd.notna(item['relative_price_rating']):
row['sub_factor']['relative_price_rating'] = item['relative_price_rating']
if 'avg_cost_rating' in item and pd.notna(item['avg_cost_rating']):
row['sub_factor']['avg_cost_rating'] = item['avg_cost_rating']
if 'total_price_score_minmax' in item and pd.notna(item['total_price_score_minmax']):
row['sub_factor']['total_price_score'] = item['total_price_score_minmax']
if 'estate_charge_score_minmax' in item and pd.notna(item['estate_charge_score_minmax']):
row['sub_factor']['estate_charge_score'] = item['estate_charge_score_minmax']
if 'relative_price_ranking_score_minmax' in item and pd.notna(item['relative_price_ranking_score_minmax']):
row['sub_factor']['relative_price_score'] = item['relative_price_ranking_score_minmax']
if 'avg_cost_score_minmax' in item and pd.notna(item['avg_cost_score_minmax']):
row['sub_factor']['avg_cost_score'] = item['avg_cost_score_minmax']
action = {'_op_type': 'update', # 操作 index update create delete action = {'_op_type': 'update', # 操作 index update create delete
'_index': index_name, # index '_index': index_name, # index
@@ -156,58 +293,58 @@ def sync_by_city(city_now):
if len(community_summary_df) == 0: if len(community_summary_df) == 0:
log.warning(f"{city_now}无法评级,数量低于5") log.warning(f"{city_now}无法评级,数量低于5")
return return
#plot_img(community_summary_df)
save_png(community_summary_df) save_png(community_summary_df)
nextcloudutils.up_to_nextcloud(save_png_path + "/" + city_now + ".png", nextcloud_aim_file_path + "/" + city_now + ".png") nextcloudutils.up_to_nextcloud(save_png_path + "/" + city_now + ".png", nextcloud_aim_file_path + "/" + city_now + ".png")
def clear_rating(temp_df):
"""
清空评级信息
"""
actions = list()
count = 0
for index, item in temp_df.iterrows():
_id = str(item["_id"])
action = {'_op_type': 'update', # 操作 index update create delete
'_index': index_name, # index
'_id': _id, # 请勿忘记ID
'doc': {'consumption_rating': None, 'consumption_potential': None},
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("清理中: 当前已清理{}".format(count))
if len(actions) > 0:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("清理完毕: 共清理{}".format(count))
def sync(city=None): def sync(city=None):
clear_sql = "select city, region, name from community_summary where consumption_potential is not null and (total_price is null or estate_charge is null or avg_cost is null) "
if pd.isna(city): if pd.isna(city):
# 清空缺少因子的老潜力
temp_df = es206_client.scan_by_sql_to_data_frame(clear_sql)
clear_rating(temp_df)
run = False
for city_now in reasonable_df['city'].unique(): for city_now in reasonable_df['city'].unique():
#if pd.isna(reasonable_dict[city_now]): #if pd.isna(reasonable_dict[city_now]):
sync_by_city(city_now) # if city_now == '昌都':
# run = True
# if run:
sync_by_city(city_now)
else: else:
clear_sql += f" and city='{city}' "
temp_df = es206_client.scan_by_sql_to_data_frame(clear_sql)
clear_rating(temp_df)
sync_by_city(city) sync_by_city(city)
def filter_min_max(community_summary_df):
community_summary_df['normal_consumption_potential'] = QuantileTransformer(output_distribution="normal").fit_transform(community_summary_df[['consumption_potential']])
# total_price5 = np.percentile(a=community_summary_df['total_price'], q=2, axis=0)
# total_price95 = np.percentile(a=community_summary_df['total_price'], q=98, axis=0)
# community_summary_df.loc[community_summary_df['total_price'] > total_price95, 'total_price'] = total_price95
# community_summary_df.loc[community_summary_df['total_price'] < total_price5, 'total_price'] = total_price5
#
# estate_charge5 = np.percentile(a=community_summary_df['estate_charge'], q=2, axis=0)
# estate_charge95 = np.percentile(a=community_summary_df['estate_charge'], q=98, axis=0)
# community_summary_df.loc[community_summary_df['estate_charge'] > estate_charge95, 'estate_charge'] = estate_charge95
# community_summary_df.loc[community_summary_df['estate_charge'] < estate_charge5, 'estate_charge'] = estate_charge5
#
# avg_cost5 = np.percentile(a=community_summary_df['avg_cost'], q=2, axis=0)
# avg_cost95 = np.percentile(a=community_summary_df['avg_cost'], q=98, axis=0)
# community_summary_df.loc[community_summary_df['avg_cost'] > avg_cost95, 'avg_cost'] = avg_cost95
# community_summary_df.loc[community_summary_df['avg_cost'] < avg_cost5, 'avg_cost'] = avg_cost5
#if city in factor_dict:
# 户均总价按极值设置可靠性不高,当户均总价补充后,图中大于最大值的量非常多
# community_summary_df.loc[
# community_summary_df['total_price'] < factor_dict[city]['min_total_price'], 'total_price'] = \
# factor_dict[city]['min_total_price']
# community_summary_df.loc[
# community_summary_df['total_price'] > factor_dict[city]['max_total_price'], 'total_price'] = \
# factor_dict[city]['max_total_price']
# community_summary_df.loc[
# community_summary_df['estate_charge'] < factor_dict[city]['min_estate_charge'], 'estate_charge'] = \
# factor_dict[city]['min_estate_charge']
# community_summary_df.loc[
# community_summary_df['estate_charge'] > factor_dict[city]['max_estate_charge'], 'estate_charge'] = \
# factor_dict[city]['max_estate_charge']
# community_summary_df.loc[community_summary_df['avg_cost'] < factor_dict[city]['min_avg_cost'], 'avg_cost'] = \
# factor_dict[city]['min_avg_cost']
# community_summary_df.loc[community_summary_df['avg_cost'] > factor_dict[city]['max_avg_cost'], 'avg_cost'] = \
# factor_dict[city]['max_avg_cost']
return community_summary_df
def save_png(community_city): def save_png(community_city):
""" """
生成财力评级图表 生成财力评级图表
@@ -251,6 +388,47 @@ def save_png(community_city):
plt.close() plt.close()
city = sys.argv[1] if len(sys.argv) > 1 else None def plot_img(community_city):
sync(city) """
生成财力评级图表
"""
curr_city = community_city['city'].unique()[0]
# 解决中文显示问题
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False
plt.title(curr_city)
plt.subplot(3, 1, 1)
community_city['consumption_score'].hist(bins=100)
plt.xlabel('consumption_rating')
plt.tight_layout()
plt.subplot(3, 2, 3)
community_city['total_price_score_minmax'].hist(bins=100)
plt.xlabel('total_price')
plt.tight_layout()
plt.subplot(3, 2, 4)
community_city['estate_charge_score_minmax'].hist(bins=100)
plt.xlabel('estate_charge')
plt.tight_layout()
plt.subplot(3, 2, 5)
community_city['avg_cost_score_minmax'].hist(bins=100)
plt.xlabel('avg_cost')
plt.tight_layout()
plt.subplot(3, 2, 6)
community_city['relative_price_ranking_score_minmax'].hist(bins=100)
plt.xlabel('relative_price_ranking')
plt.tight_layout()
# 关窗口
plt.show()
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
#dingutils.send_text("【分布图生成完毕】财力评级已完成, 请点击 https://nextcloud.fangjia.com/s/Jqs5BzXi8xLYeH9 查看分布图", "c7ac6c86bfbe49a95e30a68616c4d784ae7781c715430dc9c5f6b46d3e6ebd78") #dingutils.send_text("【分布图生成完毕】财力评级已完成, 请点击 https://nextcloud.fangjia.com/s/Jqs5BzXi8xLYeH9 查看分布图", "c7ac6c86bfbe49a95e30a68616c4d784ae7781c715430dc9c5f6b46d3e6ebd78")
@@ -0,0 +1,181 @@
"""
财力评级统计
"""
from elasticsearch import helpers
from goojia_utils import ElasticSearchClient, dateutils, nextcloudutils, dingutils, data_utils
import pandas as pd
import numpy as np
import logging
import os
import sys
log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
save_path_prefix = "/home/airflow/data/community"
save_png_path = save_path_prefix + "/consumption_rating"
if not os.path.exists(save_png_path):
os.makedirs(save_png_path)
nextcloud_aim_file_path = '财力评级/财力评级/财力评级统计报表'
def format_score_range(min_score, max_score):
"""格式化分数范围,防止Excel自动转换"""
if pd.isna(min_score) or pd.isna(max_score):
return "-"
# 使用Excel公式语法强制识别为文本
return f'="{int(min_score)}-{int(max_score)}"'
def stat(community_summary_df):
df = community_summary_df
grouped = df.groupby(["city"])
results = []
for city, group in grouped:
# 总数
total_num = len(group)
# 财力评级不为空数量
group = group.dropna(subset=['consumption_rating'])
rating_num = len(group)
# 计算财力评级指标
rating_counts = group["consumption_rating"].value_counts(normalize=True)
rating_1 = rating_counts.get(1, 0)
rating_2 = rating_counts.get(2, 0)
rating_3 = rating_counts.get(3, 0)
rating_4 = rating_counts.get(4, 0)
rating_5 = rating_counts.get(5, 0)
rating_6 = rating_counts.get(6, 0)
# 计算分数范围
rating_1_scores = group[(group["consumption_rating"] == 1) & (group['estate_charge_score'].notna()) & (group['total_price_score'].notna()) & (group['relative_price_score'].notna()) & (group['avg_cost_score'].notna())]["consumption_score"]
rating_2_scores = group[(group["consumption_rating"] == 2) & (group['estate_charge_score'].notna()) & (group['total_price_score'].notna()) & (group['relative_price_score'].notna()) & (group['avg_cost_score'].notna())]["consumption_score"]
rating_3_scores = group[(group["consumption_rating"] == 3) & (group['estate_charge_score'].notna()) & (group['total_price_score'].notna()) & (group['relative_price_score'].notna()) & (group['avg_cost_score'].notna())]["consumption_score"]
rating_4_scores = group[(group["consumption_rating"] == 4) & (group['estate_charge_score'].notna()) & (group['total_price_score'].notna()) & (group['relative_price_score'].notna()) & (group['avg_cost_score'].notna())]["consumption_score"]
rating_5_scores = group[(group["consumption_rating"] == 5) & (group['estate_charge_score'].notna()) & (group['total_price_score'].notna()) & (group['relative_price_score'].notna()) & (group['avg_cost_score'].notna())]["consumption_score"]
rating_6_scores = group[(group["consumption_rating"] == 6) & (group['estate_charge_score'].notna()) & (group['total_price_score'].notna()) & (group['relative_price_score'].notna()) & (group['avg_cost_score'].notna())]["consumption_score"]
# 计算置信度占比
confidence_counts = group["credit"].value_counts(normalize=True)
results.append({
"城市": city,
"楼盘总量": total_num,
"有财力评级数量": rating_num,
"财力评级1级占比": round(rating_1 * 100, 1),
"财力评级1级分数范围": format_score_range(rating_1_scores.min(), rating_1_scores.max()) if not rating_1_scores.empty else "-",
"财力评级2级占比": round(rating_2 * 100, 1),
"财力评级2级分数范围": format_score_range(rating_2_scores.min(), rating_2_scores.max()) if not rating_2_scores.empty else "-",
"财力评级3级占比": round(rating_3 * 100, 1),
"财力评级3级分数范围": format_score_range(rating_3_scores.min(), rating_3_scores.max()) if not rating_3_scores.empty else "-",
"财力评级4级占比": round(rating_4 * 100, 1),
"财力评级4级分数范围": format_score_range(rating_4_scores.min(), rating_4_scores.max()) if not rating_4_scores.empty else "-",
"财力评级5级占比": round(rating_5 * 100, 1),
"财力评级5级分数范围": format_score_range(rating_5_scores.min(), rating_5_scores.max()) if not rating_5_scores.empty else "-",
"财力评级6级占比": round(rating_6 * 100, 1),
"财力评级6级分数范围": format_score_range(rating_6_scores.min(), rating_6_scores.max()) if not rating_6_scores.empty else "-",
"高置信度占比": round(confidence_counts.get("高", 0) * 100, 1),
"中置信度占比": round(confidence_counts.get("中", 0) * 100, 1),
"低置信度占比": round(confidence_counts.get("低", 0) * 100, 1)
})
# 生成最终报表
report_df = pd.DataFrame(results)
aim_file = dateutils.format_date(dateutils.localize()) + '财力评级统计结果.csv'
report_df.to_csv(save_png_path + '/' + aim_file, index=False)
nextcloudutils.up_to_nextcloud(save_png_path + "/" + aim_file, nextcloud_aim_file_path + "/" + aim_file)
def detail(city):
"""
计算消费潜力
"""
if pd.notna(city):
if ',' in city:
cities = city.split(',')
else:
cities = [city]
query = {
"bool": {
"must": [
{
"terms": {
"city": cities
}
}
]
}
}
else:
query = {
"bool": {
"must": [
{
"match_all": {}
}
]
}
}
community_summary_df = es206_client.scan_to_data_frame_with_source(query={"query": query}, source=[
"id",
"city",
"region",
"name",
"avg_price",
"total_price",
"estate_charge",
"estate_charge_fix",
"estate_charge_flag",
"main_house_area",
"relative_price_ranking",
"avg_cost",
"consumption_rating",
"price_rating", "avg_price_calc_type",
"consumption_score",
"sub_factor.total_price_rating",
"sub_factor.total_price_score",
"sub_factor.estate_charge_score",
"sub_factor.relative_price_score",
"sub_factor.avg_cost_score",
"credit_detail.total_price_score"
], index="community_summary")
# 减少重复查询
community_summary_df = data_utils.flatten_json(community_summary_df, 'credit_detail')
community_summary_df = community_summary_df.rename(columns={'total_price_score': 'credit_detail_total_price_score'})
community_summary_df = data_utils.flatten_json(community_summary_df, 'sub_factor')
community_summary_df['diff'] = np.abs(community_summary_df['consumption_rating'] - community_summary_df['price_rating'])
community_summary_df['total_price_diff'] = np.abs(community_summary_df['consumption_rating'] - community_summary_df['total_price_rating'])
community_summary_df['estate_charge_type'] = np.where(
pd.notna(community_summary_df['estate_charge']), # 仅当 estate_charge 不为空时处理
np.where(
community_summary_df['estate_charge_flag'] == 1,
'周边',
'自有'
),
''
)
community_summary_df['credit'] = np.where(community_summary_df['total_price_diff'] == 0, '高', np.where(community_summary_df['total_price_diff'] == 1, '中', np.where(community_summary_df['total_price_diff'] >= 2, '低', '-')))
stat(community_summary_df)
community_summary_df = community_summary_df.rename(columns={'city': '城市', 'region': '区域', 'name': '小区名称', 'consumption_score': '财力评分', 'consumption_rating': '财力评级',
'total_price_rating': '户均总价评级', 'total_price_diff': '差值',
'total_price': '户均总价', 'avg_price': '小区均价', 'main_house_area': '典型面积', 'relative_price_ranking': '相对价格排名',
'avg_cost': '人均消费', 'estate_charge': '物业费', 'estate_charge_fix': '物业费修正', 'price_rating': '价格评级', 'estate_charge_type': '物业费获取方式',
'credit': '置信度', 'total_price_score': '户均总价评分', 'estate_charge_score': '物业费评分', 'relative_price_score': '相对价格评分', 'avg_cost_score': '人均消费评分'
})
community_summary_df['计算方式'] = np.where(community_summary_df['avg_price_calc_type'] == 1,'指导价',np.where(community_summary_df['avg_price_calc_type'] == 2,'自身样本',np.where(community_summary_df['avg_price_calc_type'] == 3,'第三方',
np.where(community_summary_df['avg_price_calc_type'] == 4,'交易数据',np.where(community_summary_df['avg_price_calc_type'] == 5,'周边楼盘',np.where(community_summary_df['avg_price_calc_type'] == 6,'推导', np.where(community_summary_df['avg_price_calc_type'] == 7,'周边推算','无方式')))))))
community_summary_df['获取方式'] = np.where(community_summary_df['credit_detail_total_price_score'] <= 6,'周边','自有')
aim_file = dateutils.format_date(dateutils.localize()) + '财力评级结果.csv'
community_summary_df.to_csv(save_png_path + '/' + aim_file, index=False, columns=["城市", "区域", "小区名称", "财力评分", "财力评级", "户均总价评级", "差值", "户均总价", "小区均价", "计算方式", "典型面积",
"获取方式", "相对价格排名", "人均消费", "物业费", "物业费获取方式", '置信度', '户均总价评分', '物业费评分', '相对价格评分', '人均消费评分'])
nextcloudutils.up_to_nextcloud(save_png_path + "/" + aim_file, nextcloud_aim_file_path + "/" + aim_file)
dingutils.send_text("【统计】财力评级统计完毕,请点击 https://nextcloud.fangjia.com/s/nMbzk6ZKJDimapf 查看", "32400b949005817550937f82a82c064fe5be65c23204306442c15e8b485d7ef2")
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
detail(sys_city)
@@ -0,0 +1,83 @@
"""
置信度及总分计算
"""
import logging
import pandas as pd
from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, dateutils, data_utils
from elasticsearch import helpers
import sys
log = logging.getLogger(__name__)
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
path = "/home/airflow/data/community"
m_date = dateutils.iso_format()
avg_price_map = dict()
def sync(city=None):
log.info(f"开始置信度打分 {index_name}")
# 加载库中均价数据
query_filter = [{"range": {"consumption_rating": {"gt": 0}}}]
if pd.notna(city):
query_filter.append({"term": {"city": city}})
query = {
"_source": ["city", "region", "credit_detail"],
"query": {
"bool": {
"filter": query_filter
}
}
}
community_summary_df = es206_client.scan_to_data_frame("community_summary", query)
community_summary_df = data_utils.flatten_json(community_summary_df, 'credit_detail')
community_summary_df['total_price_score'].fillna(0, inplace=True)
community_summary_df['relative_price_ranking_score'].fillna(0, inplace=True)
community_summary_df['estate_charge_score'].fillna(0, inplace=True)
community_summary_df['total_score'] = round(0.6 * community_summary_df['total_price_score'] + 0.2 * community_summary_df['relative_price_ranking_score'] + 0.2 * community_summary_df['estate_charge_score'])
community_summary_df['credit'] = round(community_summary_df['total_score'] / 10, 2)
actions = list()
count = 0
for index, item in community_summary_df[community_summary_df['total_score'].notna()].iterrows():
doc_info = {
'credit': item['credit'],
'credit_detail': {
'total_score': item['total_score']
}
}
action = {
"_op_type": "update",
"_index": index_name,
"_id": item['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
logging.info("已更新:" + str(count))
actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
logging.info("已更新:" + str(count))
actions.clear()
log.info(f"完成置信度打分 {index_name}")
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -0,0 +1,112 @@
"""
同步删除小区基本信息至ES206
"""
# 兼容性补丁:解决 Elasticsearch 8.x/9.x 客户端在老版本 ES 集群上因 compatible-with 头部导致的 ApiError(406)
try:
import elasticsearch
ver = getattr(elasticsearch, '__version__', None)
if ver and not str(ver).startswith('7.'):
original_init = elasticsearch.Elasticsearch.__init__
def patched_init(self, *args, **kwargs):
headers = kwargs.get('headers', {})
headers['Accept'] = 'application/json'
headers['Content-Type'] = 'application/json'
kwargs['headers'] = headers
original_init(self, *args, **kwargs)
elasticsearch.Elasticsearch.__init__ = patched_init
except Exception:
pass
import logging
import pandas as pd
from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, dateutils
from elasticsearch import helpers
import os
import sys
log = logging.getLogger(__name__)
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
m_date = dateutils.iso_format()
path = "/home/airflow/data/community"
if not os.path.exists(path):
os.makedirs(path)
def deal_comm(community_summary_df, seaweed_df):
"""
删除额外小区信息
"""
actions = list()
diff_ids = set(community_summary_df['_id']).difference(set(seaweed_df['_id']))
for _id in diff_ids:
action = {
"_op_type": "delete",
"_index": index_name,
"_id": _id,
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
actions.clear()
def sync(city=None):
log.info(f"开始删除已删除小区至206 {index_name}...")
seaweed_query = {'status': 1, 'cat': 'district'}
if pd.notna(city):
seaweed_query['city'] = city
seaweed_df = pd.DataFrame(
list(seaweed_conn.find(seaweed_query, {'city': 1, 'region': 1, 'name': 1, 'status': 1}).batch_size(100000)))
seaweed_df['_id'] = seaweed_df['_id'].astype(str)
sql = f"select id from {index_name} "
if pd.notna(city):
sql += f" where city='{city}'"
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
if len(community_summary_df) > 0:
community_summary_df['flag'] = 1
seaweed_new_df = pd.merge(seaweed_df, community_summary_df, how='left', on=['_id'])
seaweed_existed_df = seaweed_new_df[seaweed_new_df['flag'] == 1]
else:
seaweed_existed_df = pd.DataFrame()
actions = list()
# 对于存在的删除
count = 0
for index, item in seaweed_existed_df.iterrows():
action = {
"_op_type": "delete",
"_index": index_name,
"_id": str(item['_id'])
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("删除:" + str(count))
actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("删除:" + str(count))
actions.clear()
log.info(f"完成删除小区基本属性至206 {index_name}")
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+17 -9
View File
@@ -3,14 +3,15 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock from threading import Lock
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -32,13 +33,15 @@ def deal(actions, item):
doc_info = dict() doc_info = dict()
# 幼儿园 # 幼儿园
kindergarten = poiutils.find_around_by_location(curr_city, 1000, [lng, lat], "学校", "幼儿园") # kindergarten = poiutils.find_around_by_location(curr_city, 1000, [lng, lat], "学校", "幼儿园")
kindergarten = poiutils.find_around_by_location_from_mongo(curr_city, 1000, lng, lat, "学校", "幼儿园")
doc_info['kindergarten_num'] = len(kindergarten) doc_info['kindergarten_num'] = len(kindergarten)
if len(kindergarten) > 0: if len(kindergarten) > 0:
doc_info['kindergarten'] = ",".join(kindergarten) doc_info['kindergarten'] = ",".join(kindergarten)
# 大学 # 大学
university = poiutils.find_around_by_location(curr_city, 5000, [lng, lat], "学校", "大学") # university = poiutils.find_around_by_location(curr_city, 5000, [lng, lat], "学校", "大学")
university = poiutils.find_around_by_location_from_mongo(curr_city, 1000, lng, lat, "学校", "大学")
doc_info['university_num'] = len(university) doc_info['university_num'] = len(university)
if len(university) > 0: if len(university) > 0:
doc_info['university'] = ",".join(university) doc_info['university'] = ",".join(university)
@@ -63,10 +66,10 @@ def deal(actions, item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步教育资源【幼儿园、大学】至206 {index_name}") log.info(f"开始同步教育资源【幼儿园、大学】至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
actions = list() actions = list()
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
@@ -83,5 +86,10 @@ def sync(city=None):
log.info(f"完成同步教育资源至206 {index_name}") log.info(f"完成同步教育资源至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+21 -11
View File
@@ -3,14 +3,15 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock from threading import Lock
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -32,8 +33,10 @@ def deal(actions, item):
doc_info = dict() doc_info = dict()
# 小学 # 小学
key_primary_school = poiutils.find_around_by_location(curr_city, 1000, [lng, lat], "学校", "小学", "重点") # key_primary_school = poiutils.find_around_by_location(curr_city, 1000, [lng, lat], "学校", "小学", "重点")
ordinary_primary_school = poiutils.find_around_by_location(curr_city, 1000, [lng, lat], "学校", "小学", "普通") key_primary_school = poiutils.find_around_by_location_from_mongo(curr_city, 1000, lng, lat, "学校", "小学", "重点")
# ordinary_primary_school = poiutils.find_around_by_location(curr_city, 1000, [lng, lat], "学校", "小学", "普通")
ordinary_primary_school = poiutils.find_around_by_location_from_mongo(curr_city, 1000, lng, lat, "学校", "小学","普通")
doc_info['key_primary_school_num'] = len(key_primary_school) doc_info['key_primary_school_num'] = len(key_primary_school)
if len(key_primary_school) > 0: if len(key_primary_school) > 0:
@@ -44,8 +47,10 @@ def deal(actions, item):
doc_info['ordinary_primary_school'] = ",".join(ordinary_primary_school) doc_info['ordinary_primary_school'] = ",".join(ordinary_primary_school)
# 中学 # 中学
key_middle_school = poiutils.find_around_by_location(curr_city, 2000, [lng, lat], "学校", "中学", "重点") # key_middle_school = poiutils.find_around_by_location(curr_city, 2000, [lng, lat], "学校", "中学", "重点")
ordinary_middle_school = poiutils.find_around_by_location(curr_city, 2000, [lng, lat], "学校", "中学", "普通") key_middle_school = poiutils.find_around_by_location_from_mongo(curr_city, 1000, lng, lat, "学校", "中学", "重点")
# ordinary_middle_school = poiutils.find_around_by_location(curr_city, 2000, [lng, lat], "学校", "中学", "普通")
ordinary_middle_school = poiutils.find_around_by_location_from_mongo(curr_city, 1000, lng, lat, "学校", "中学","普通")
doc_info['key_middle_school_num'] = len(key_middle_school) doc_info['key_middle_school_num'] = len(key_middle_school)
if len(key_middle_school) > 0: if len(key_middle_school) > 0:
@@ -76,10 +81,10 @@ def deal(actions, item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步教育资源【小学、中学】至206 {index_name}") log.info(f"开始同步教育资源【小学、中学】至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
actions = list() actions = list()
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
@@ -96,5 +101,10 @@ def sync(city=None):
log.info(f"完成同步教育资源至206 {index_name}") log.info(f"完成同步教育资源至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -7,18 +7,73 @@ import logging
import sys import sys
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
file_path = "/opt/airflow/dags/repo/dags/business/business/community/data"
mutex = Lock()
estate_charge_min_max_df = pd.read_excel(file_path + '/全国各区域物业费中位值.xlsx')
estate_charge_city_region_dict = dict()
def save(community_summary_df): # 加载各区域物业费最大最小值
if 'estate_charge' not in community_summary_df.columns: def deal_estate_charge(x):
city_now = x['city']
region_now = x['region']
if pd.notna(city_now) and pd.notna(region_now):
estate_charge_city_region_dict[city_now + region_now] = {'min': x['3'], 'max': x['97']}
elif pd.notna(city_now):
estate_charge_city_region_dict[city_now] = {'min': x['3'], 'max': x['97']}
estate_charge_min_max_df.apply(lambda x: deal_estate_charge(x), axis=1)
def save_fix(community_summary_df):
if 'estate_charge_fix' not in community_summary_df.columns:
log.info("该城市无调整物业费")
return
df = community_summary_df[['_id', 'city', 'estate_charge_fix', 'estate_charge_score']]
df = df.dropna(subset=['estate_charge_fix'])
actions = list()
count = 0
# 批量更新
m_date = dateutils.iso_format()
for index, item in df.iterrows():
row = dict()
_id = str(item["_id"])
row['estate_charge_fix'] = item['estate_charge_fix']
row['credit_detail'] = {'estate_charge_score': int(item['estate_charge_score'])}
row['m_date'] = m_date
action = {'_op_type': 'update',
'_index': index_name,
'_id': _id, # 请勿忘记ID
'doc': row,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("已更新{}".format(count))
if len(actions) > 0:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("更新完成,共更新: {}".format(count))
return df
def save_fill(community_summary_df):
if 'estate_charge' not in community_summary_df.columns or 'estate_charge_score' not in community_summary_df.columns:
log.info("该城市无补充物业费") log.info("该城市无补充物业费")
return return
df = community_summary_df[['_id', 'city', 'estate_charge']] df = community_summary_df[['_id', 'city', 'estate_charge', 'estate_charge_score']]
df = df.dropna(subset=['estate_charge']) df = df.dropna(subset=['estate_charge'])
actions = list() actions = list()
count = 0 count = 0
@@ -29,6 +84,8 @@ def save(community_summary_df):
row = dict() row = dict()
_id = str(item["_id"]) _id = str(item["_id"])
row['estate_charge'] = item['estate_charge'] row['estate_charge'] = item['estate_charge']
if pd.notna(item['estate_charge_score']):
row['credit_detail'] = {'estate_charge_score': int(item['estate_charge_score'])}
# 1 表示为物业费补充的,在实际应用中不输出 # 1 表示为物业费补充的,在实际应用中不输出
row['estate_charge_flag'] = 1 row['estate_charge_flag'] = 1
row['m_date'] = m_date row['m_date'] = m_date
@@ -57,8 +114,8 @@ def find_nearest_by_location(city, location):
docs = list() docs = list()
try: try:
scan = es206_client.scan(index_name, { scan = es206_client.scan(index_name, {
"size": 3, "size": 4,
"_source": ["finish_building_date", "estate_charge"], "_source": ["finish_building_date", "estate_charge", "estate_charge_fix"],
"query": { "query": {
"bool": { "bool": {
"filter": [ "filter": [
@@ -88,7 +145,12 @@ def find_nearest_by_location(city, location):
} }
] ]
}, preserve_order=True) }, preserve_order=True)
ignored_first = False
for doc in scan: for doc in scan:
# 忽略自己
if ignored_first is False:
ignored_first = True
continue
docs.append(doc['_source']) docs.append(doc['_source'])
if len(docs) == 3: if len(docs) == 3:
return pd.DataFrame(docs) return pd.DataFrame(docs)
@@ -98,7 +160,7 @@ def find_nearest_by_location(city, location):
return None return None
def get_estate_charge(finish_building_date, docs): def get_estate_charge_for_adjust(finish_building_date, docs):
aim_year = int(finish_building_date / 10) aim_year = int(finish_building_date / 10)
for index, item in docs.iterrows(): for index, item in docs.iterrows():
if pd.notna(item['finish_building_date']) and pd.notna(item['estate_charge']) and aim_year == int(item['finish_building_date'] / 10): if pd.notna(item['finish_building_date']) and pd.notna(item['estate_charge']) and aim_year == int(item['finish_building_date'] / 10):
@@ -106,31 +168,153 @@ def get_estate_charge(finish_building_date, docs):
return None return None
def sync_by_city(city_now): def get_estate_charge(finish_building_date, docs):
log.info(f"{city_now}开始补充物业费....") aim_year = int(finish_building_date / 10)
sql = "select id, city, region, name, estate_charge, finish_building_date, location from community_summary where city='" + city_now + "' and (estate_charge is null or estate_charge_flag=1)" for index, item in docs.iterrows():
if pd.notna(item['finish_building_date']) and pd.notna(item['estate_charge_fix']) and aim_year == int(item['finish_building_date'] / 10):
return item['estate_charge_fix']
return None
find_count = 0
def deal_adjust(community):
global find_count
find_count += 1
if find_count % 1000 == 0:
log.info("已查找" + str(find_count))
max_now = 7
city_short = community['city']
region_short = community['region']
if (city_short + region_short) in estate_charge_city_region_dict:
max_now = estate_charge_city_region_dict[city_short + region_short]['max']
elif city_short in estate_charge_city_region_dict:
max_now = estate_charge_city_region_dict[city_short]['max']
if community['estate_charge'] > max_now:
estate_charge, score = find_estate_charge_for_adjust(community, max_now)
if pd.notna(estate_charge):
mutex.acquire(timeout=60)
community['estate_charge_fix'] = estate_charge
community['estate_charge_score'] = score
mutex.release()
def adjust_estate_charge(city_now):
log.info(f"{city_now}开始调整物业费....")
sql = f"select id, city, region, name, estate_charge, finish_building_date, location from community_summary where city='{city_now}' and estate_charge_flag != 1 and estate_type != '别墅' "
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql) community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
with ThreadPoolExecutor(max_workers=50) as pool: community_summary_list = list()
def deal(index, item): community_summary_df.apply(lambda x: community_summary_list.append(dict(x)), axis=1)
if index % 1000 == 0 and index > 0:
log.info(f"已查找{index}") with ThreadPoolExecutor(max_workers=30) as pool:
if pd.notna(item['location']) and pd.notna(item['location']['lon']): for community_summary in community_summary_list:
location = [item['location']['lon'], item['location']['lat']] pool.submit(deal_adjust, community_summary)
docs = find_nearest_by_location(city_now, location) # for community_summary in community_summary_list:
estate_charge = None # deal_adjust(community_summary)
if len(docs) <= 0 or 'estate_charge' not in docs.columns: community_summary_df = pd.DataFrame(community_summary_list)
return save_fix(community_summary_df)
if pd.notna(item['finish_building_date']) and 'finish_building_date' in docs.columns:
# 根据相近建筑年代赋值 log.info(f"{city_now}完成调整物业费....")
estate_charge = get_estate_charge(item['finish_building_date'], docs)
if pd.isna(estate_charge):
# 取周边均值 def find_estate_charge_for_adjust(community, max_now):
estate_charge = docs['estate_charge'].mean() """
if pd.notna(estate_charge): 查找小区周边物业费
community_summary_df.loc[index, "estate_charge"] = estate_charge """
for index, item in community_summary_df.iterrows(): estate_charge = None
pool.submit(deal, index, item) if pd.isna(community['location']) or pd.isna(community['location']['lon']):
save(community_summary_df) return estate_charge
location = [community['location']['lon'], community['location']['lat']]
docs = find_nearest_by_location(community['city'], location)
if len(docs) <= 0 or 'estate_charge' not in docs.columns:
return estate_charge
# 记录打分
score = None
if pd.notna(community['finish_building_date']) and 'finish_building_date' in docs.columns:
# 根据相近建筑年代赋值
estate_charge = get_estate_charge_for_adjust(community['finish_building_date'], docs)
score = 9
if pd.isna(estate_charge) or estate_charge > max_now:
# 取周边均值
estate_charge = docs['estate_charge'].mean()
score = 8
# 如果查找的还大于分位值,不调整
if estate_charge > max_now:
estate_charge = None
score = None
return estate_charge, score
def find_estate_charge(community, max_now):
"""
查找小区周边物业费
"""
estate_charge = None
if pd.isna(community['location']) or pd.isna(community['location']['lon']):
return estate_charge, 5
location = [community['location']['lon'], community['location']['lat']]
docs = find_nearest_by_location(community['city'], location)
if len(docs) <= 0 or 'estate_charge' not in docs.columns:
return estate_charge, 5
if 'estate_charge_fix' not in docs.columns:
docs['estate_charge_fix'] = None
# 用修正后的物业费补充
docs['estate_charge_fix'].fillna(docs['estate_charge'], inplace=True)
score = None
if pd.notna(community['finish_building_date']) and 'finish_building_date' in docs.columns:
# 根据相近建筑年代赋值
estate_charge = get_estate_charge(community['finish_building_date'], docs)
score = 7
if pd.isna(estate_charge) or estate_charge > max_now:
# 取周边均值
estate_charge = docs['estate_charge_fix'].mean()
score = 6
if estate_charge > max_now:
estate_charge = None
score = None
return estate_charge, score
find_count = 0
def deal_fill(community):
global find_count
find_count += 1
if find_count % 1000 == 0:
log.info("已查找" + str(find_count))
max_now = 7
city_short = community['city']
region_short = community['region']
if (city_short + region_short) in estate_charge_city_region_dict:
max_now = estate_charge_city_region_dict[city_short + region_short]['max']
elif city_short in estate_charge_city_region_dict:
max_now = estate_charge_city_region_dict[city_short]['max']
if community['estate_type'] == '别墅':
# 别墅不限制物业费最大值
max_now = 99
estate_charge, score = find_estate_charge(community, max_now)
if pd.notna(estate_charge):
mutex.acquire(timeout=60)
community["estate_charge"] = estate_charge
community["estate_charge_score"] = score
mutex.release()
def fill_estate_charge(city_now):
log.info(f"{city_now}开始补充物业费....")
sql = "select id, city, region, name, estate_charge, estate_charge_fix, finish_building_date, estate_type, location from community_summary where city='" + city_now + "' and (estate_charge is null or estate_charge_flag=1) "
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
community_summary_list = list()
community_summary_df.apply(lambda x: community_summary_list.append(dict(x)), axis=1)
with ThreadPoolExecutor(max_workers=30) as pool:
for community_summary in community_summary_list:
pool.submit(deal_fill, community_summary)
community_summary_df = pd.DataFrame(community_summary_list)
save_fill(community_summary_df)
log.info(f"{city_now}完成补充物业费....")
def sync_by_city(city_now):
adjust_estate_charge(city_now)
fill_estate_charge(city_now)
def sync(city=None): def sync(city=None):
@@ -141,5 +325,10 @@ def sync(city=None):
sync_by_city(city) sync_by_city(city)
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -14,7 +14,7 @@ from sklearn.preprocessing import QuantileTransformer
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
path = "/opt/airflow/dags/repo/dags/business/business/community/data" path = "/opt/airflow/dags/repo/dags/business/business/community/data"
@@ -82,11 +82,11 @@ def calc_estate_charge_rating(community_summary_df):
actions = list() actions = list()
count = 0 count = 0
if len(df) < 5: if len(df) < 6:
# 对于样本低于5的直接排名 # 对于样本低于5的直接排名
df[target] = df['estate_charge'].rank(method='dense', ascending=False).astype(int) df[target] = df['estate_charge'].rank(method='dense', ascending=False).astype(int)
else: else:
df = gmm_train(5, df, source, target) df = gmm_train(6, df, source, target)
# 批量更新 # 批量更新
for index, item in df.iterrows(): for index, item in df.iterrows():
row = dict() row = dict()
@@ -164,6 +164,11 @@ def save_png(community_city):
plt.close() plt.close()
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
#dingutils.send_text("【分布图生成完毕】物业费评级已完成, 请点击 https://nextcloud.fangjia.com/s/HbifLoEDc5wi8HR 查看分布图", "c7ac6c86bfbe49a95e30a68616c4d784ae7781c715430dc9c5f6b46d3e6ebd78") #dingutils.send_text("【分布图生成完毕】物业费评级已完成, 请点击 https://nextcloud.fangjia.com/s/HbifLoEDc5wi8HR 查看分布图", "c7ac6c86bfbe49a95e30a68616c4d784ae7781c715430dc9c5f6b46d3e6ebd78")
+8 -3
View File
@@ -17,7 +17,7 @@ from threading import Lock
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -141,5 +141,10 @@ def sync(city=None):
log.info(f"完成同步附加标签至206 {index_name}") log.info(f"完成同步附加标签至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -0,0 +1,73 @@
"""
省市区简拼转全拼
"""
from goojia_utils import ElasticSearchClient, dateutils
import logging
import pandas as pd
from elasticsearch import helpers
import sys
log = logging.getLogger(__name__)
es89_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
path = "/home/airflow/data/community"
m_date = dateutils.iso_format()
count = 0
short_df = es89_client.scan_by_sql_to_data_frame("select province, city, city_short, district, district_short from administrative_mapping where type='区域' ")
short_map = {}
def deal_short_map(x):
short_map[x['city_short'] + x['district_short']] = x['province'] + '|' + x['city'] + '|' + x['district']
short_df.apply(lambda x: deal_short_map(x), axis=1)
def deal(actions, x):
global count
doc_info = dict()
if pd.notna(x['city']) and pd.notna(x['region']):
if x['city'] + x['region'] in short_map:
short_info = short_map[x['city'] + x['region']]
doc_info['province_full'] = short_info.split('|')[0]
doc_info['city_full'] = short_info.split('|')[1]
doc_info['region_full'] = short_info.split('|')[2]
action_doc = {
"_op_type": "update",
"_index": index_name,
"_id": x['_id'],
"doc": doc_info
}
if len(doc_info) > 0:
actions.append(action_doc)
if len(actions) == 5000:
helpers.bulk(es89_client.elastic_client, actions)
count += len(actions)
log.info(f"已更新{count}")
actions.clear()
def sync(city=None):
log.info(f"开始转换省市区全拼至206 {index_name}")
sql = "select city, region from community_summary"
if pd.notna(city):
sql += " where city='" + city + "'"
community_summary_df = es89_client.scan_by_sql_to_data_frame(sql)
actions = list()
community_summary_df.apply(lambda x: deal(actions, x), axis=1)
if len(actions) > 0:
helpers.bulk(es89_client.elastic_client, actions)
actions.clear()
log.info(f"完成转换省市区全拼至206 {index_name}")
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+13 -7
View File
@@ -3,14 +3,15 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock from threading import Lock
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -77,10 +78,10 @@ def deal(actions, item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步生活配套至206 {index_name}") log.info(f"开始同步生活配套至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
actions = list() actions = list()
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
@@ -97,5 +98,10 @@ def sync(city=None):
log.info(f"完成同步生活配套至206 {index_name}") log.info(f"完成同步生活配套至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+17 -10
View File
@@ -3,7 +3,7 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, dateutils, areautils, poiutils from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, dateutils, poiutils
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock from threading import Lock
@@ -11,7 +11,8 @@ import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed') seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -32,13 +33,15 @@ def deal(actions, item):
doc_info = dict() doc_info = dict()
# 三甲 # 三甲
top_three_hospital = poiutils.find_around_by_location(curr_city, 5000, [lng, lat], "医院", "三甲医院") # top_three_hospital = poiutils.find_around_by_location(curr_city, 5000, [lng, lat], "医院", "三甲医院")
top_three_hospital = poiutils.find_around_by_location_from_mongo(curr_city, 5000, lng, lat, "医院", None,"三甲")
doc_info['top_three_hospital_num'] = len(top_three_hospital) doc_info['top_three_hospital_num'] = len(top_three_hospital)
if len(top_three_hospital) > 0: if len(top_three_hospital) > 0:
doc_info['top_three_hospital'] = ",".join(top_three_hospital) doc_info['top_three_hospital'] = ",".join(top_three_hospital)
# 非三甲 # 非三甲
non_3a_hospital = poiutils.find_around_by_location(curr_city, 5000, [lng, lat], "医院", "普通医院") # non_3a_hospital = poiutils.find_around_by_location(curr_city, 5000, [lng, lat], "医院", "普通医院")
non_3a_hospital = poiutils.find_around_by_location_from_mongo(curr_city, 5000, lng, lat, "医院", None,"普通")
doc_info['non_3a_hospital_num'] = len(non_3a_hospital) doc_info['non_3a_hospital_num'] = len(non_3a_hospital)
if len(non_3a_hospital) > 0: if len(non_3a_hospital) > 0:
doc_info['non_3a_hospital'] = ",".join(non_3a_hospital) doc_info['non_3a_hospital'] = ",".join(non_3a_hospital)
@@ -63,11 +66,10 @@ def deal(actions, item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步医院至206 {index_name}") log.info(f"开始同步医院至206 {index_name}")
seaweed_query = {'status': 0, 'cat': 'district'} query = {}
if pd.notna(city): if pd.notna(city):
seaweed_query['city'] = city query['city'] = city
seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8')
actions = list() actions = list()
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
@@ -84,5 +86,10 @@ def sync(city=None):
log.info(f"完成同步医院至206 {index_name}") log.info(f"完成同步医院至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+9 -4
View File
@@ -7,7 +7,7 @@ import pandas as pd
from elasticsearch import helpers from elasticsearch import helpers
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -19,7 +19,7 @@ count = 0
def deal(actions, x): def deal(actions, x):
global count global count
doc_info = dict() doc_info = dict()
if pd.notna(x['avg_price']) and pd.notna(x['rent_price']): if 'rent_price' in x and pd.notna(x['avg_price']) and pd.notna(x['rent_price']):
doc_info['sell_to_rent'] = int(x['avg_price'] / x['rent_price']) doc_info['sell_to_rent'] = int(x['avg_price'] / x['rent_price'])
if pd.notna(x['household_count']): if pd.notna(x['household_count']):
doc_info['person_num'] = int(x['household_count'] * 2.3) doc_info['person_num'] = int(x['household_count'] * 2.3)
@@ -57,5 +57,10 @@ def sync(city=None):
log.info(f"完成同步售租比、人口数至206 {index_name}") log.info(f"完成同步售租比、人口数至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -0,0 +1,129 @@
"""
百分位计算 1 最好 100最差
"""
from goojia_utils import ElasticSearchClient
import pandas as pd
import logging
from elasticsearch import helpers
import sys
log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
def calc_percentile(city=None):
"""
计算消费等级
"""
sql = "select id, city, region, name, avg_price, total_price, avg_cost, rent_price, relative_price_ranking, finish_building_date, estate_charge, estate_charge_fix, household_count, build_area from community_summary"
if pd.notna(city):
sql += " where city='" + city + "'"
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
if 'estate_charge_fix' not in community_summary_df.columns:
community_summary_df['estate_charge_fix'] = 0
community_summary_df['estate_charge_fix'] = community_summary_df['estate_charge'].combine_first(community_summary_df['estate_charge_fix'])
community_summary_df['relative_price_ranking'] = community_summary_df['relative_price_ranking'] * 100
if 'rent_price' not in community_summary_df.columns:
community_summary_df['rent_price'] = 0
consumption_aim_df = community_summary_df[['id', 'city', 'avg_price', 'total_price', 'avg_cost', 'rent_price', 'relative_price_ranking', 'finish_building_date',
'estate_charge_fix', 'household_count', 'build_area']]
number = []
for i in range(1, 101):
number.append(i / 100)
percents_actions = list()
# 计算百分位的字段
columns = ['avg_price', 'total_price', 'avg_cost', 'rent_price', 'relative_price_ranking', 'finish_building_date',
'estate_charge_fix', 'household_count', 'build_area']
for city_now, item in consumption_aim_df.groupby(by=['city']).describe(percentiles=number).iterrows():
for column in columns:
for i in range(1, 101):
if str(i) + '%' in item[column]:
if pd.notna((item[column])[str(i) + '%']):
action = {'city': city_now, 'type': column,
'price': int((item[column])[str(i) + '%']), 'percent': i}
else:
if pd.notna((item[column])[str(i) + '.0%']):
action = {'city': city_now, 'type': column,
'price': int((item[column])[str(i) + '.0%']), 'percent': i}
percents_actions.append(action)
percents_df = pd.DataFrame(percents_actions)
cities = consumption_aim_df['city'].unique()
actions = list()
count = 0
for city_now in cities:
log.info("开始计算:{}".format(city_now))
df = consumption_aim_df[consumption_aim_df['city'] == city_now].copy()
percents_now_df = percents_df[percents_df['city'] == city_now].copy()
try:
for i in range(1, 101):
if i == 1:
for column in columns:
df.loc[(df[column] <= percents_now_df[(percents_now_df['type'] == column) & (percents_now_df['percent'] == i)].iloc[0]['price']), column + '_percentile'] = 101 - i
else:
for column in columns:
df.loc[(df[column] > percents_now_df[(percents_now_df['type'] == column) & (percents_now_df['percent'] == i - 1)].iloc[0]['price']) & (df[column] <= percents_now_df[(percents_now_df['type'] == column) & (percents_now_df['percent'] == i)].iloc[0]['price']), column + '_percentile'] = 101 - i
except Exception as e:
log.error("解析错误{}", e)
continue
# 批量更新
for index, item in df.iterrows():
row = dict()
_id = str(item["id"])
for column in columns:
if pd.notna(item[column + '_percentile']):
if 'estate_charge_fix' == column:
db_column = "estate_charge_percentile"
else:
db_column = column + '_percentile'
row[db_column] = int(item[column + '_percentile'])
# 根据百分位计算小区规模,优先户数,然后根据建筑面积
if 'household_count_percentile' in item and pd.notna(item['household_count_percentile']):
if item['household_count_percentile'] <= 33:
row['scale'] = '小'
elif item['household_count_percentile'] <= 66:
row['scale'] = '中'
else:
row['scale'] = '大'
if 'scale' not in row:
if 'build_area_percentile' in item and pd.notna(item['build_area_percentile']):
if item['build_area_percentile'] <= 33:
row['scale'] = '小'
elif item['build_area_percentile'] <= 66:
row['scale'] = '中'
else:
row['scale'] = '大'
if 'build_area_percentile' in row:
del row['build_area_percentile']
action = {'_op_type': 'update',
'_index': index_name,
'_id': _id, # 请勿忘记ID
'doc': row,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("已更新:{}".format(count))
if len(actions) > 0:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("更新完成,共更新: {}".format(count))
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
calc_percentile(city)
else:
calc_percentile(sys_city)
@@ -0,0 +1,84 @@
"""
人口评级
"""
from goojia_utils import ElasticSearchClient
import pandas as pd
import logging
from elasticsearch import helpers
import sys
from sklearn.mixture import GaussianMixture
log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
def gmm_train(total_rate, consumption_df, source, target):
# 高斯混合模型
gmm = GaussianMixture(n_components=total_rate, covariance_type='tied', max_iter=200, n_init=10).fit(consumption_df[[source]])
consumption_df.loc[:, 'cluster'] = gmm.predict(consumption_df[[source]])
consumption_df.loc[:, 'price_mean_cluster'] = consumption_df.groupby('cluster')[source].transform('mean')
consumption_df.loc[:, target] = consumption_df['price_mean_cluster'].rank(method='dense', ascending=False).astype(int)
consumption_df = consumption_df.drop(columns=['cluster', 'price_mean_cluster'], axis=1)
return consumption_df
def calc_person_rating(city=None):
"""
计算消费等级
"""
sql = "select id, city, region, name, household_count from community_summary"
if pd.notna(city):
sql += " where city='" + city + "'"
community_summary_df = es206_client.scan_by_sql_to_data_frame(sql)
consumption_aim_df = community_summary_df[['_id', 'city', 'household_count']]
consumption_aim_df = consumption_aim_df.dropna(subset=['household_count'])
cities = consumption_aim_df['city'].unique()
source = "household_count"
target = "person_rating"
actions = list()
count = 0
for city in cities:
df = consumption_aim_df[consumption_aim_df['city'] == city].copy()
if len(df) < 5:
continue
df = gmm_train(5, df, source, target)
# 批量更新
for index, item in df.iterrows():
row = dict()
_id = str(item["_id"])
row['person_rating'] = item['person_rating']
action = {'_op_type': 'update',
'_index': index_name,
'_id': _id, # 请勿忘记ID
'doc': row}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("评级中: 当前已评级{}".format(count))
if len(actions) > 0:
helpers.bulk(client=es206_client.elastic_client, actions=actions)
count += len(actions)
actions.clear()
log.info("评级完成,共更新: {}".format(count))
def sync(city=None):
calc_person_rating(city)
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+35 -21
View File
@@ -3,12 +3,13 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils, areautils from goojia_utils import ElasticSearchClient, commutils, dateutils, areautils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -35,14 +36,15 @@ def build_doc(item):
doc_dict['rent_price'] = int(item['rent_price']) doc_dict['rent_price'] = int(item['rent_price'])
if 'guide_price' in item and pd.notna(item['guide_price']): if 'guide_price' in item and pd.notna(item['guide_price']):
doc_dict['guide_price'] = int(item['guide_price']) doc_dict['guide_price'] = int(item['guide_price'])
if 'avg_price_calc_type' in item and pd.notna(item['avg_price_calc_type']):
doc_dict['avg_price_calc_type'] = int(item['avg_price_calc_type'])
return doc_dict return doc_dict
def sync_by_city(city, total_seaweed_df): def sync_by_city(city):
seaweed_df = total_seaweed_df[total_seaweed_df['city'] == city].copy() seaweed_df = pd.DataFrame(list(community_summary_conn.find({"city": city}, {"_id": 1, "city": 1, "region": 1, "name": 1})))
if len(seaweed_df) == 0: if len(seaweed_df) == 0:
return return
seaweed_df = seaweed_df.reset_index(drop=True)
# 小区所在城市均价 # 小区所在城市均价
city_price_df = commutils.get_city_price(city=city) city_price_df = commutils.get_city_price(city=city)
@@ -61,16 +63,17 @@ def sync_by_city(city, total_seaweed_df):
# 小区均价 # 小区均价
avg_price_df = commutils.get_avg_price(city=city) avg_price_df = commutils.get_avg_price(city=city)
if len(avg_price_df) > 0: if len(avg_price_df) > 0:
avg_price_df = avg_price_df[['city', 'district_name', 'region', 'avg_price']] avg_price_df = avg_price_df[['city', 'district_name', 'region', 'avg_price', 'value_type']]
avg_price_df = avg_price_df.rename(columns={'district_name': 'name'}) avg_price_df = avg_price_df.rename(columns={'district_name': 'name', 'value_type': 'avg_price_calc_type'})
seaweed_df = pd.merge(seaweed_df, avg_price_df, how='left', on=['city', 'region', 'name']) seaweed_df = pd.merge(seaweed_df, avg_price_df, how='left', on=['city', 'region', 'name'])
# 小区均价区域排名 # 小区均价区域排名
seaweed_df['avg_price_region_ranking'] = \ seaweed_df['avg_price_region_ranking'] = \
seaweed_df.groupby(by=['city', 'region']).rank(method='first', na_option='bottom', ascending=False)['avg_price'] seaweed_df.groupby(by=['city', 'region']).rank(method='first', na_option='bottom', ascending=False)[
'avg_price']
# 小区均价城市排名 # 小区均价城市排名
seaweed_df['avg_price_ranking'] = \ seaweed_df['avg_price_ranking'] = \
seaweed_df.groupby(by=['city']).rank(method='first', na_option='bottom', ascending=False)['avg_price'] seaweed_df.groupby(by=['city']).rank(method='first', na_option='bottom', ascending=False)['avg_price']
# 12个月均价涨跌幅 # 12个月均价涨跌幅
avg_price_up_and_downs_df = commutils.get_avg_price_up_and_downs(cities=[city], months=-12) avg_price_up_and_downs_df = commutils.get_avg_price_up_and_downs(cities=[city], months=-12)
@@ -78,16 +81,22 @@ def sync_by_city(city, total_seaweed_df):
avg_price_up_and_downs_df = avg_price_up_and_downs_df[['city', 'region', 'district_name', 'up_and_downs']] avg_price_up_and_downs_df = avg_price_up_and_downs_df[['city', 'region', 'district_name', 'up_and_downs']]
avg_price_up_and_downs_df = avg_price_up_and_downs_df.rename( avg_price_up_and_downs_df = avg_price_up_and_downs_df.rename(
columns={'district_name': 'name', 'up_and_downs': 'year_on_year'}) columns={'district_name': 'name', 'up_and_downs': 'year_on_year'})
avg_price_up_and_downs_df = avg_price_up_and_downs_df.drop_duplicates(subset=['city', 'region', 'name'], keep='first').reset_index(
drop=True).copy()
seaweed_df = pd.merge(seaweed_df, avg_price_up_and_downs_df, how='left', on=['city', 'region', 'name']) seaweed_df = pd.merge(seaweed_df, avg_price_up_and_downs_df, how='left', on=['city', 'region', 'name'])
# 近半年均价涨跌幅 # 近半年均价涨跌幅
avg_price_up_and_downs_df = commutils.get_avg_price_up_and_downs(cities=[city], months=-6) avg_price_up_and_downs_df = commutils.get_avg_price_up_and_downs(cities=[city], months=-6)
if len(avg_price_up_and_downs_df) > 0: if len(avg_price_up_and_downs_df) > 0:
avg_price_up_and_downs_df = avg_price_up_and_downs_df[['city', 'region', 'district_name', 'up_and_downs']] avg_price_up_and_downs_df = avg_price_up_and_downs_df[['city', 'region', 'district_name', 'up_and_downs']]
avg_price_up_and_downs_df = avg_price_up_and_downs_df.rename( avg_price_up_and_downs_df = avg_price_up_and_downs_df.rename(
columns={'district_name': 'name', 'up_and_downs': 'half_on_year'}) columns={'district_name': 'name', 'up_and_downs': 'half_on_year'})
avg_price_up_and_downs_df = avg_price_up_and_downs_df.drop_duplicates(subset=['city', 'region', 'name'], keep='first').reset_index(
drop=True).copy()
seaweed_df = pd.merge(seaweed_df, avg_price_up_and_downs_df, how='left', on=['city', 'region', 'name']) seaweed_df = pd.merge(seaweed_df, avg_price_up_and_downs_df, how='left', on=['city', 'region', 'name'])
# 平米租金 # 平米租金
rent_price_df = commutils.get_rent_price(city=city) rent_price_df = commutils.get_rent_price(city=city)
if len(rent_price_df) > 0: if len(rent_price_df) > 0:
@@ -101,7 +110,6 @@ def sync_by_city(city, total_seaweed_df):
gov_price_df = gov_price_df[['city', 'region', 'name', 'avg_price']] gov_price_df = gov_price_df[['city', 'region', 'name', 'avg_price']]
gov_price_df = gov_price_df.rename(columns={'avg_price': 'guide_price'}) gov_price_df = gov_price_df.rename(columns={'avg_price': 'guide_price'})
seaweed_df = pd.merge(seaweed_df, gov_price_df, how='left', on=['city', 'region', 'name']) seaweed_df = pd.merge(seaweed_df, gov_price_df, how='left', on=['city', 'region', 'name'])
actions = list() actions = list()
count = 0 count = 0
for index, item in seaweed_df.iterrows(): for index, item in seaweed_df.iterrows():
@@ -113,9 +121,11 @@ def sync_by_city(city, total_seaweed_df):
"_id": item['_id'], "_id": item['_id'],
"doc": doc_info "doc": doc_info
} }
actions.append(action) actions.append(action)
if len(actions) == 5000: if len(actions) == 5000:
print(index)
print(item)
helpers.bulk(es206_client.elastic_client, actions) helpers.bulk(es206_client.elastic_client, actions)
count += len(actions) count += len(actions)
log.info("更新:" + str(count)) log.info("更新:" + str(count))
@@ -125,20 +135,24 @@ def sync_by_city(city, total_seaweed_df):
actions.clear() actions.clear()
def sync(city=None): def sync(curr_city=None):
log.info(f"开始同步价格信息至206 {index_name}") log.info(f"开始同步价格信息至206 {index_name}")
if pd.notna(city): if pd.notna(curr_city):
cities = [city] cities = [curr_city]
else: else:
cities = areautils.get_all_cities() cities = areautils.get_all_cities()
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') for city_now in cities:
for city in cities: log.info(f"开始同步{city_now}")
log.info(f"开始同步{city}") sync_by_city(city_now)
sync_by_city(city, seaweed_df) log.info(f"完成同步{city_now}")
log.info(f"完成同步{city}")
log.info(f"完成同步价格信息至206 {index_name}") log.info(f"完成同步价格信息至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -14,7 +14,7 @@ from sklearn.mixture import GaussianMixture
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
@@ -66,7 +66,8 @@ def calc_price_rating(city=None):
action = {'_op_type': 'update', action = {'_op_type': 'update',
'_index': index_name, '_index': index_name,
'_id': _id, # 请勿忘记ID '_id': _id, # 请勿忘记ID
'doc': row} 'doc': row,
"_retry_on_conflict": 5}
actions.append(action) actions.append(action)
if len(actions) == 5000: if len(actions) == 5000:
helpers.bulk(client=es206_client.elastic_client, actions=actions) helpers.bulk(client=es206_client.elastic_client, actions=actions)
@@ -84,5 +85,10 @@ def sync(city=None):
calc_price_rating(city) calc_price_rating(city)
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -4,20 +4,23 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils from goojia_utils import ElasticSearchClient, commutils, dateutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
import sys import sys
from threading import Lock
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
m_date = dateutils.iso_format() m_date = dateutils.iso_format()
mutex = Lock()
def find_around_seaweed(lng, lat): def find_around_seaweed(city_now, lng, lat):
""" """
查找周边均价 查找周边均价
""" """
@@ -28,6 +31,13 @@ def find_around_seaweed(lng, lat):
"query": { "query": {
"bool": { "bool": {
"filter": [ "filter": [
{
"term": {
"city": {
"value": city_now
}
}
},
{ {
"range": { "range": {
"avg_price": { "avg_price": {
@@ -50,68 +60,106 @@ def find_around_seaweed(lng, lat):
def build_doc(item): def build_doc(item):
doc_dict = dict() doc_dict = dict()
if 'relative_price_ranking' in item and pd.notna(item['relative_price_ranking']): if 'relative_price_ranking' in item:
doc_dict['relative_price_ranking'] = item['relative_price_ranking'] if pd.notna(item['relative_price_ranking']):
doc_dict['relative_price_ranking'] = item['relative_price_ranking']
else:
doc_dict['relative_price_ranking'] = None
if 'credit_detail' in item:
doc_dict['credit_detail'] = item['credit_detail']
return doc_dict return doc_dict
find_count = 0
actions = list()
def find_around_price_item(seaweed):
if 'lng' not in seaweed:
return
lng = commutils.format_lat_lng(seaweed['lng'])
lat = commutils.format_lat_lng(seaweed['lat'])
if pd.notna(lng):
city_now = seaweed['city']
around_seaweed_df = find_around_seaweed(city_now, lng, lat)
# 某些经纬度根本不在该城市下,数据问题
if len(around_seaweed_df) == 0:
return
global find_count
find_count += 1
if find_count % 1000 == 0:
log.info("已查找" + str(find_count))
mutex.acquire()
if len(around_seaweed_df) > 1:
around_seaweed_df['relative_price_ranking'] = around_seaweed_df['avg_price'].rank(pct=True)
temp_df = around_seaweed_df[around_seaweed_df['id'] == seaweed['_id']]
if len(temp_df) > 0:
seaweed['relative_price_ranking'] = temp_df.iloc[0]['relative_price_ranking']
else:
seaweed['relative_price_ranking'] = None
elif len(around_seaweed_df) == 1:
seaweed['relative_price_ranking'] = 0.5
around_community_num = len(around_seaweed_df)
if around_community_num >= 20:
relative_price_ranking_score = 10
elif 5 <= around_community_num < 20:
relative_price_ranking_score = 9
elif 1 < around_community_num < 5:
relative_price_ranking_score = 8
elif around_community_num == 1:
relative_price_ranking_score = 7
seaweed['credit_detail'] = {'around_community_num': around_community_num, 'relative_price_ranking_score': relative_price_ranking_score}
doc_info = build_doc(seaweed)
action = {
"_op_type": "update",
"_index": index_name,
"_id": seaweed['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
log.info("更新:" + str(find_count))
actions.clear()
mutex.release()
def sync(city=None): def sync(city=None):
log.info("开始同步相对价格排名数据...") log.info("开始同步相对价格排名数据...")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
def find_around_price_item(index, item): seaweed_list = list()
if index % 1000 == 0: seaweed_df.apply(lambda x: seaweed_list.append(dict(x)), axis=1)
log.info("已处理" + str(index))
if 'lng' not in item:
return
lng = commutils.format_lat_lng(item['lng'])
lat = commutils.format_lat_lng(item['lat'])
if pd.notna(lng):
around_seaweed_df = find_around_seaweed(lng, lat)
if len(around_seaweed_df) > 1:
around_seaweed_df['relative_price_ranking'] = around_seaweed_df['avg_price'].rank(pct=True)
seaweed_df.loc[index, 'relative_price_ranking'] = around_seaweed_df.loc[around_seaweed_df['id'] == item['_id']]['relative_price_ranking'].tolist()[0]
elif len(around_seaweed_df) == 1:
seaweed_df.loc[index, 'relative_price_ranking'] = 0.5
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
for index, item in seaweed_df.iterrows(): for seaweed in seaweed_list:
pool.submit(find_around_price_item, index, item) pool.submit(find_around_price_item, seaweed)
actions = list() # for seaweed in seaweed_list:
count = 0 # find_around_price_item(seaweed)
for index, item in seaweed_df.iterrows():
doc_info = build_doc(item)
action = {
"_op_type": "update",
"_index": index_name,
"_id": item['_id'],
"doc": doc_info
}
actions.append(action)
if len(actions) == 5000:
try:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新:" + str(count))
except Exception as e:
log.error(e)
actions.clear()
if len(actions) > 0: if len(actions) > 0:
try: helpers.bulk(es206_client.elastic_client, actions)
helpers.bulk(es206_client.elastic_client, actions)
except Exception as e:
log.error(e)
actions.clear() actions.clear()
log.info("完成同步相对价格排名数据...") log.info("完成同步相对价格排名数据...")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -0,0 +1,75 @@
"""
修正相对价格排名
"""
import logging
import pandas as pd
from goojia_utils import ElasticSearchClient, dateutils, data_utils
from elasticsearch import helpers
import sys
from threading import Lock
log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary"
path = "/home/airflow/data/community"
m_date = dateutils.iso_format()
mutex = Lock()
def sync(city=None):
log.info("开始调整相对价格排名数据...")
query_filter = []
if pd.notna(city):
query_filter.append({"term": {"city": city}})
query = {
"_source": ["city", "region", "relative_price_ranking", "credit_detail"],
"query": {
"bool": {
"filter": query_filter
}
}
}
community_summary_origin_df = es206_client.scan_to_data_frame("community_summary", query)
community_summary_df = community_summary_origin_df[community_summary_origin_df['credit_detail'].notna()].reset_index(drop=True)
community_summary_df = data_utils.flatten_json(community_summary_df, 'credit_detail')
community_summary_df['around_community_max_num'] = community_summary_df.groupby(by=['city'])['around_community_num'].transform('max')
community_summary_df['relative_price_ranking_fix'] = community_summary_df['around_community_num'] / community_summary_df['around_community_max_num'] * community_summary_df['relative_price_ranking']
actions = list()
count = 0
for index, item in community_summary_df[community_summary_df['relative_price_ranking_fix'].notna()].iterrows():
doc_info = {'relative_price_ranking_fix': round(item['relative_price_ranking_fix'], 6)}
action = {
"_op_type": "update",
"_index": index_name,
"_id": item['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新:" + str(count))
actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新:" + str(count))
actions.clear()
log.info("完成调整相对价格排名数据...")
sys_city = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+21 -10
View File
@@ -7,14 +7,17 @@ from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, date
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
import sys import sys
from threading import Lock
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed') seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
m_date = dateutils.iso_format() m_date = dateutils.iso_format()
mutex = Lock()
def build_doc(item): def build_doc(item):
@@ -30,10 +33,10 @@ def build_doc(item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步出售信息至206 {index_name}") log.info(f"开始同步出售信息至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
with ThreadPoolExecutor(30) as pool: with ThreadPoolExecutor(30) as pool:
def deal(index, item): def deal(index, item):
@@ -50,12 +53,15 @@ def sync(city=None):
names_str = "'" + names_str + "'" names_str = "'" + names_str + "'"
# 近三个月挂牌量 # 近三个月挂牌量
# sample_num = commutils.get_sample(curr_city, curr_region, names_str, cat='sell', daysBefore=93) sample_num = commutils.get_sample(curr_city, curr_region, names_str, cat='sell', daysBefore=93)
# seaweed_df.loc[index, 'three_months_sell_num'] = sample_num
# 近半年挂牌量 # 近半年挂牌量
sample_num = commutils.get_sample(curr_city, curr_region, names_str, cat='sell', daysBefore=186) sample_half_year_num = commutils.get_sample(curr_city, curr_region, names_str, cat='sell', daysBefore=186)
seaweed_df.loc[index, 'half_on_year_sell_num'] = sample_num
mutex.acquire()
seaweed_df.loc[index, 'three_months_sell_num'] = sample_num
seaweed_df.loc[index, 'half_on_year_sell_num'] = sample_half_year_num
mutex.release()
for index, item in seaweed_df.iterrows(): for index, item in seaweed_df.iterrows():
pool.submit(deal, index, item) pool.submit(deal, index, item)
@@ -85,5 +91,10 @@ def sync(city=None):
log.info(f"完成同步出售信息至206 {index_name}") log.info(f"完成同步出售信息至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -0,0 +1,231 @@
import os
import sys
import logging
import pandas as pd
from datetime import datetime
from pymongo import UpdateOne
# 兼容性补丁:解决 Elasticsearch 8.x/9.x 客户端在老版本 ES 集群上因 compatible-with 头部导致的 ApiError(406)
os.environ["ELASTIC_CLIENT_APIVERSIONING"] = "0"
try:
import elasticsearch
ver = getattr(elasticsearch, '__version__', None)
if ver and not str(ver).startswith('7.'):
original_init = elasticsearch.Elasticsearch.__init__
def patched_init(self, *args, **kwargs):
headers = kwargs.get('headers', {})
headers['Accept'] = 'application/json'
headers['Content-Type'] = 'application/json'
kwargs['headers'] = headers
original_init(self, *args, **kwargs)
elasticsearch.Elasticsearch.__init__ = patched_init
except Exception:
pass
from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
log = logging.getLogger(__name__)
# 初始化 ES 206 客户端与 85 上的 MongoDB 集合
es206_client = ElasticSearchClient(host='192.168.0.206')
average_statistics_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'average_statistics')
index_name = "community"
# 映射关系:将 ES 中的 root 主域映射为 Mongo 表中的对应二手房价格 (_esf_price) 字段
ROOT_FIELD_MAP = {
"anjuke.com": "anjuke_esf_price", # 安居客
"lianjia.com": "lianjia_esf_price", # 链家
"ke.com": "beike_esf_price", # 贝壳
"beike.com": "beike_esf_price", # 贝壳
"fang.com": "soufang_esf_price", # 房天下/搜房
"soufang.com": "soufang_esf_price", # 搜房
"5i5j.com": "5i5j_esf_price", # 我爱我家
"centanet.com": "zhongyuan_esf_price", # 中原地产
"qfang.com": "qfang_esf_price", # Q房网
"leyoujia.com": "leyoujia_esf_price", # 乐有家
"zhuge.com": "zhugezhaofang_esf_price", # 诸葛找房
"zhugezhaofang.com": "zhugezhaofang_esf_price",
"jiwu.com": "jiwu_esf_price", # 吉屋
"house365.com": "365house_esf_price", # 365淘房
"365.com": "365house_esf_price",
"fanggugu.com": "fanggugu_esf_price", # 房估估
"yungujia.com": "yungujia_esf_price", # 云估价
"tuboshi.com": "tuboshi_esf_price", # 兔博士
"xiaozijia.com": "xiaozijia_esf_price", # 小资家
"fangdoudou.com": "fangdoudou_esf_price",# 房兜兜
"fangtan.com": "fangtan_esf_price", # 房探
"guoxinda.com": "gxd_esf_price", # 国信达
"gxd.com": "gxd_esf_price",
"xingfuli.com": "xingfuli_esf_price", # 幸福里
"maitian.cn": "maitian_esf_price", # 麦田房产
"maitian.com": "maitian_esf_price",
"sina.com.cn": "sina_esf_price", # 新浪二手房
"sina.com": "sina_esf_price",
"cric.com": "cric_esf_price", # 克而瑞
"ganji.com": "ganji_esf_price", # 赶集网
}
def parse_s_date(m_date_str):
"""
将 ES 抓取时间 '2022-06-17T13:12:47.148Z' 转化为整型 s_date YYYYMM (例如 202206)
"""
try:
if pd.isna(m_date_str) or not isinstance(m_date_str, str):
return None
return int(m_date_str[:7].replace('-', ''))
except Exception:
return None
def sync_data(target_month=None):
"""
数据流式拉取、转换与批量 Upsert
"""
log.info("开始从 ES 206 的 community 索引拉取小区价格数据...")
# 构造 SQL,默认仅同步近 3 个月(90 天)内更新的数据,降低传输和存储开销
if target_month is not None:
try:
month_str = str(target_month)
year = int(month_str[:4])
month = int(month_str[4:6])
start_date = f"{year:04d}-{month:02d}-01T00:00:00"
if month == 12:
end_date = f"{year + 1:04d}-01-01T00:00:00"
else:
end_date = f"{year:04d}-{month + 1:02d}-01T00:00:00"
date_filter = f"and m_date >= '{start_date}' and m_date < '{end_date}'"
log.info(f"指定历史月份跑数,过滤条件: {start_date} 至 {end_date}")
except Exception as e:
log.error(f"解析指定月份失败,退回到默认的近3个月同步。Error: {e}")
from datetime import datetime, timedelta
three_months_ago = (datetime.now() - timedelta(days=90)).strftime('%Y-%m-%dT%H:%M:%S')
date_filter = f"and m_date >= '{three_months_ago}'"
else:
from datetime import datetime, timedelta
three_months_ago = (datetime.now() - timedelta(days=90)).strftime('%Y-%m-%dT%H:%M:%S')
date_filter = f"and m_date >= '{three_months_ago}'"
log.info(f"增量运行,过滤近3个月内有更新的数据 (m_date >= '{three_months_ago}')")
sql = f"select city, region, name, avg_price, root, m_date from {index_name} where avg_price is not null and avg_price > 0 {date_filter}"
# 城市手动过滤(可选,通过外部参数调度)
sys_city = sys.argv[1] if len(sys.argv) > 1 and not sys.argv[1].isdigit() else None
if pd.notna(sys_city):
sql += f" and city = '{sys_city}'"
log.info(f"正在过滤目标城市: {sys_city}")
df = es206_client.scan_by_sql_to_data_frame(sql)
if df is None or len(df) == 0:
log.info("未从 ES 中获取到可用数据。")
return
log.info(f"成功读取到 {len(df)} 条原始记录。开始解析字段,并根据唯一键 (city, region, name, s_date) 进行内存合并...")
# 聚合内存合并,以支持多个不同 root (渠道来源) 的价格在同一个 (城市-行政区-小区名-月份) 唯一键文档中合并设置
grouped = {}
for idx, row in df.iterrows():
city = row.get('city')
region = row.get('region')
name = row.get('name')
avg_price = row.get('avg_price')
root = row.get('root')
m_date = row.get('m_date')
if pd.isna(city) or pd.isna(region) or pd.isna(name) or pd.isna(avg_price) or pd.isna(root):
continue
try:
price_val = int(float(avg_price))
except (ValueError, TypeError):
continue
# 过滤价格 <= 0 的异常/无效数据
if price_val <= 0:
continue
s_date = parse_s_date(m_date)
if s_date is None:
continue
# 若指定了目标月份过滤(如仅同步指定月份的数据)
if target_month is not None and s_date != int(target_month):
continue
field_name = ROOT_FIELD_MAP.get(root)
if not field_name:
continue
key = (city, region, name, s_date)
if key not in grouped:
grouped[key] = {}
# 多个相同 root 冲突时,此处保留最新或较高价格(以整型存储)
current_val = grouped[key].get(field_name)
if current_val is None or price_val > current_val:
grouped[key][field_name] = price_val
log.info(f"内存聚合清洗完毕。共获得 {len(grouped)} 组唯一索引项。开始构建批量 MongoDB UpdateOne upsert 任务...")
# 构建 Bulk Write Upsert 任务
actions = []
now = datetime.now()
for (city, region, name, s_date), fields_dict in grouped.items():
# 需要 $set 写入的所有第三方价格字段及最近修改时间
update_doc = {**fields_dict}
update_doc["m_date"] = now # ModificationDate 对应实体属性
query = {
"city": city,
"region": region,
"name": name,
"s_date": s_date
}
# Upsert 动作
action = UpdateOne(
query,
{
"$set": update_doc,
"$setOnInsert": {
"category": "district", # 默认对应小区类别 district
"date": now # 对应首建/大日期 longDate
}
},
upsert=True
)
actions.append(action)
total_actions = len(actions)
if total_actions == 0:
log.info("过滤处理后无数据需写入 MongoDB。")
return
log.info(f"开始向 192.168.0.85 数据库的 average_statistics 集合中分批写入 {total_actions} 条大批处理记录...")
# 5000条一笔批量执行写入,防止网络长连接超时或内存剧增
batch_size = 5000
for i in range(0, total_actions, batch_size):
batch_actions = actions[i:i + batch_size]
result = average_statistics_conn.bulk_write(batch_actions)
log.info(f"批次 {i // batch_size + 1} 写入情况: "
f"Matched={result.matched_count}, "
f"Upserted={result.upserted_count}, "
f"Modified={result.modified_count}")
log.info("数据同步流任务全部成功结束!")
if __name__ == '__main__':
# 支持传入参数如 `python sync_es_price_to_mongo.py 202401` 进行指定月份跑数
month_arg = None
if len(sys.argv) > 1:
arg = sys.argv[1]
if arg.isdigit() and len(arg) == 6:
month_arg = int(arg)
log.info(f"读取到指定月份运行参数: {month_arg}")
sync_data(target_month=month_arg)
+16 -52
View File
@@ -3,15 +3,13 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, commutils, dateutils from goojia_utils import MAIN_MONGO_CLIENT, ElasticSearchClient, dateutils
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor
import re
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed') seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -26,59 +24,20 @@ def build_doc(item):
return doc_dict return doc_dict
# def avg_price_map_deal(x):
# avg_price_map[x['city'] + x['region'] + x['name']] = x['avg_price']
# return x
def deal(seaweed_df, index, item):
if index % 1000 == 0:
log.info("已处理" + str(index))
curr_city = item['city']
curr_region = item['region']
curr_name = item['name']
names = [curr_name]
if pd.notna(item['alias']) and '[]' != item['alias']:
alias_list = re.sub("\[|\]|'", '', item['alias']).split(',')
if len(alias_list) > 0:
for alias_item in alias_list:
names.append(alias_item)
# 户均总价
# if (curr_city + curr_region + curr_name) in avg_price_map:
# avg_price = avg_price_map[curr_city + curr_region + curr_name]
# if pd.notna(avg_price):
# 由于覆盖率较低,暂不过滤
# min_avg_price = 0.8 * avg_price
# max_avg_rice = 1.2 * avg_price
total_price = commutils.get_total_price(curr_city, curr_region, names, day_before=366)
seaweed_df.loc[index, 'total_price'] = total_price
def sync(city=None): def sync(city=None):
log.info(f"开始同步户均总价信息至206 {index_name}") log.info(f"开始同步户均总价信息至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8')
if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy()
seaweed_df = seaweed_df.reset_index(drop=True)
# 加载库中均价数据 # 加载库中均价数据
# query_sql = "SELECT city, region, name, avg_price FROM community_summary " query_sql = "SELECT city, region, name, avg_price, house_avg_area FROM community_summary WHERE avg_price > 0 and house_avg_area > 0 "
# if pd.notna(city): if pd.notna(city):
# query_sql += " WHERE city='" + city + "' " query_sql += " AND city='" + city + "' "
# community_summary_df = es206_client.scan_by_sql_to_data_frame(query_sql) community_summary_df = es206_client.scan_by_sql_to_data_frame(query_sql)
# community_summary_df.apply(lambda x: avg_price_map_deal(x), axis=1)
with ThreadPoolExecutor(30) as pool:
for index, item in seaweed_df.iterrows():
pool.submit(deal, seaweed_df, index, item)
actions = list() actions = list()
count = 0 count = 0
for index, item in seaweed_df.iterrows(): community_summary_df['total_price'] = community_summary_df['avg_price'] * community_summary_df['house_avg_area']
community_summary_df['total_price'] = community_summary_df['total_price'].astype(int)
for index, item in community_summary_df.iterrows():
doc_info = build_doc(item) doc_info = build_doc(item)
action = { action = {
@@ -107,5 +66,10 @@ def sync(city=None):
log.info(f"完成同步户均总价信息至206 {index_name}") log.info(f"完成同步户均总价信息至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
@@ -6,19 +6,24 @@ import pandas as pd
from goojia_utils import ElasticSearchClient, dateutils from goojia_utils import ElasticSearchClient, dateutils
from elasticsearch import helpers from elasticsearch import helpers
import sys import sys
from concurrent.futures import ThreadPoolExecutor
from threading import Lock
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
index_name = "community_summary" index_name = "community_summary"
path = "/opt/airflow/dags/repo/dags/business/business/community/data" path = "/opt/airflow/dags/repo/dags/business/business/community/data"
m_date = dateutils.iso_format() m_date = dateutils.iso_format()
mutex = Lock()
def build_doc(item): def build_doc(item):
doc_dict = dict() doc_dict = dict()
if 'total_price' in item and pd.notna(item['total_price']): if 'total_price' in item and pd.notna(item['total_price']):
doc_dict['total_price'] = int(item['total_price']) doc_dict['total_price'] = int(item['total_price'])
if 'total_price_score' in item and pd.notna(item['total_price_score']):
doc_dict['credit_detail'] = {'total_price_score': item['total_price_score']}
return doc_dict return doc_dict
@@ -49,13 +54,6 @@ def save(community_factor_df):
actions.clear() actions.clear()
def complete_by_area(community_df):
factor_df = pd.read_csv(path + '/小区户均总价面积因子已去极值.csv', sep='@', encoding='UTF-8')
community_factor_df = pd.merge(community_df, factor_df, how='left', on=['_id', 'city', 'region', 'name'])
community_factor_df['total_price'] = community_factor_df['avg_price'] * community_factor_df['factor']
return community_factor_df
def complete_by_build_year_area(community_factor_df): def complete_by_build_year_area(community_factor_df):
build_year_area_df = pd.read_excel(path + '/全国各区各建筑年代户均面积已去极值.xlsx') build_year_area_df = pd.read_excel(path + '/全国各区各建筑年代户均面积已去极值.xlsx')
build_year_area_dict = dict() build_year_area_dict = dict()
@@ -93,6 +91,8 @@ def complete_by_build_year_area(community_factor_df):
item['city'] + "|" + item['region'] + "|2020及以后"] if item['city'] + "|" + item[ item['city'] + "|" + item['region'] + "|2020及以后"] if item['city'] + "|" + item[
'region'] + "|2020及以后" in build_year_area_dict else None 'region'] + "|2020及以后" in build_year_area_dict else None
community_factor_df['total_price'] = community_factor_df['avg_price'] * community_factor_df['factor'] community_factor_df['total_price'] = community_factor_df['avg_price'] * community_factor_df['factor']
# 按各区各建筑年代户均面积补充, 得6分
community_factor_df.loc[community_factor_df['total_price'].notna(), 'total_price_score'] = 6
def complete_by_region_area(community_factor_df): def complete_by_region_area(community_factor_df):
@@ -105,22 +105,78 @@ def complete_by_region_area(community_factor_df):
community_factor_df['total_price'] = community_factor_df['avg_price'] * community_factor_df['factor'] community_factor_df['total_price'] = community_factor_df['avg_price'] * community_factor_df['factor']
def find_around_avg_area(city_now, region, lng, lat):
ret_es = es206_client.elastic_client.search({
"query": {
"bool": {
"filter": [
{"term": {"city": {"value": city_now}}},
{"term": {"region": {"value": region}}},
{"geo_distance": {"distance": 3000, "location": [lng, lat]}},
{"range": {"house_avg_area": {"gt": 0}}}
]
}
},
"size": 0,
"aggs": {
"house_avg_area": {
"avg": {
"field": "house_avg_area"
}
}
}
}, index='community_summary')
return ret_es['aggregations']['house_avg_area']['value']
def complete_by_round_area(community_factor_df):
"""
按周边三公里小区户均面积均值补充
"""
def deal_complete_by_round_area(index, item):
avg_price = item['avg_price']
house_avg_area = find_around_avg_area(item['city'], item['region'], item['location']['lon'],
item['location']['lat'])
mutex.acquire(timeout=60)
community_factor_df.loc[index, 'factor'] = house_avg_area
if pd.notna(avg_price) and pd.notna(house_avg_area):
community_factor_df.loc[index, 'total_price'] = avg_price * house_avg_area
# 按周边三公里小区户均面积均值补充,得5分
community_factor_df.loc[index, 'total_price_score'] = 5
mutex.release()
# for index, item in community_factor_df[community_factor_df['total_price'].isna()].iterrows():
# deal_complete_by_round_area(index, item)
with ThreadPoolExecutor(max_workers=30) as pool:
for index, item in community_factor_df[community_factor_df['total_price'].isna()].iterrows():
pool.submit(deal_complete_by_round_area, index, item)
def sync(city=None): def sync(city=None):
log.info(f"开始补全户均总价信息至206 {index_name}") log.info(f"开始补全户均总价信息至206 {index_name}")
sql = "select city, region, name, avg_price, finish_building_date from community_summary where avg_price is not null and total_price is null " sql = "select city, region, name, avg_price, finish_building_date, location from community_summary where avg_price is not null and total_price is null "
if pd.notna(city): if pd.notna(city):
sql += " and city='" + city + "'" sql += " and city='" + city + "'"
community_df = es206_client.scan_by_sql_to_data_frame(sql) community_df = es206_client.scan_by_sql_to_data_frame(sql)
if len(community_df) > 0: if len(community_df) > 0:
community_factor_df = complete_by_area(community_df) community_df['total_price'] = None
complete_by_build_year_area(community_factor_df) community_df['total_price_score'] = None
complete_by_region_area(community_factor_df) complete_by_build_year_area(community_df)
save(community_factor_df) #complete_by_region_area(community_df)
complete_by_round_area(community_df)
save(community_df)
log.info(f"完成补全户均总价信息至206 {index_name}") log.info(f"完成补全户均总价信息至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+14 -8
View File
@@ -4,11 +4,12 @@
import logging import logging
import pandas as pd import pandas as pd
import sys import sys
from goojia_utils import ElasticSearchClient, commutils, dateutils from goojia_utils import ElasticSearchClient, commutils, dateutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -26,11 +27,11 @@ def sync(city=None):
log.info(f"开始同步成交信息至206 {index_name}") log.info(f"开始同步成交信息至206 {index_name}")
cities = None cities = None
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8')
query = {}
if pd.notna(city): if pd.notna(city):
cities = [city] query['city'] = city
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
seaweed_df = seaweed_df.reset_index(drop=True)
deal_price_df = commutils.get_deal_avg_price(cities, 3) deal_price_df = commutils.get_deal_avg_price(cities, 3)
if len(deal_price_df) <= 0: if len(deal_price_df) <= 0:
@@ -75,5 +76,10 @@ def sync(city=None):
log.info(f"完成同步成交信息至206 {index_name}") log.info(f"完成同步成交信息至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+13 -7
View File
@@ -3,14 +3,15 @@
""" """
import logging import logging
import pandas as pd import pandas as pd
from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils from goojia_utils import ElasticSearchClient, commutils, dateutils, poiutils, MAIN_MONGO_CLIENT
from elasticsearch import helpers from elasticsearch import helpers
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from threading import Lock from threading import Lock
import sys import sys
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206') es206_client = ElasticSearchClient(host='192.168.0.89', username='elastic', password='elasticsearch')
community_summary_conn = MAIN_MONGO_CLIENT.get_collection('tmp', 'community_summary')
index_name = "community_summary" index_name = "community_summary"
path = "/home/airflow/data/community" path = "/home/airflow/data/community"
@@ -71,10 +72,10 @@ def deal(actions, item):
def sync(city=None): def sync(city=None):
log.info(f"开始同步交通至206 {index_name}") log.info(f"开始同步交通至206 {index_name}")
seaweed_df = pd.read_csv(path + '/seaweed.csv', sep='@', encoding='UTF-8') query = {}
if pd.notna(city): if pd.notna(city):
seaweed_df = seaweed_df[seaweed_df['city'] == city].copy() query['city'] = city
seaweed_df = seaweed_df.reset_index(drop=True) seaweed_df = pd.DataFrame(list(community_summary_conn.find(query)))
actions = list() actions = list()
with ThreadPoolExecutor(max_workers=30) as pool: with ThreadPoolExecutor(max_workers=30) as pool:
@@ -87,5 +88,10 @@ def sync(city=None):
log.info(f"完成同步交通至206 {index_name}") log.info(f"完成同步交通至206 {index_name}")
city = sys.argv[1] if len(sys.argv) > 1 else None sys_city = sys.argv[1] if len(sys.argv) > 1 else None
sync(city) if pd.notna(sys_city) and "," in sys_city:
cities = sys_city.split(",")
for city in cities:
sync(city)
else:
sync(sys_city)
+82 -76
View File
@@ -1,76 +1,82 @@
import logging # import logging
import sys # import sys
from goojia_utils import MAIN_MONGO_CLIENT, DIDATA_MONGO_CLIENT, areautils # from goojia_utils import MAIN_MONGO_CLIENT, DIDATA_MONGO_CLIENT, areautils
from pymongo import UpdateOne # from pymongo import UpdateOne
from bson.objectid import ObjectId # from bson.objectid import ObjectId
import pandas as pd # import pandas as pd
import time # import time
#
log = logging.getLogger(__name__) # log = logging.getLogger(__name__)
#
type = sys.argv[1] if len(sys.argv) > 1 else 'all' # type = sys.argv[1] if len(sys.argv) > 1 else 'all'
log.info("同步类型:" + type) # log.info("同步类型:" + type)
#
#
def sync_district_office(type): # def sync_district_office(type):
if type != 'district' and type != 'office': # if type != 'district' and type != 'office':
return # return
seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed') # seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
seaweed_didata_conn = DIDATA_MONGO_CLIENT.get_collection('fangjia', 'seaweed') # seaweed_didata_conn = DIDATA_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
#
cities = areautils.get_all_cities() # cities = areautils.get_all_cities()
update_date = int(round(time.time() * 1000)) # update_date = int(round(time.time() * 1000))
for city in cities: # for city in cities:
actions = list() # actions = list()
id_list = list() # id_list = list()
for doc in seaweed_conn.find({'city': city, 'cat': type, 'status': 0}, {'city': 1, 'region': 1, 'block': 1, 'cat': 1, 'name': 1, 'address': 1, # for doc in seaweed_conn.find({'city': city, 'cat': type, 'status': 0},
'estate_type2': 1, 'township': 1, 'alias': 1, 'addr_alias': 1, 'lng': 1, 'lat': 1, 'status': 1})\ # {'city': 1, 'region': 1, 'block': 1, 'cat': 1, 'name': 1, 'address': 1,
.batch_size(10000): # 'estate_type2': 1, 'township': 1, 'alias': 1, 'addr_alias': 1, 'lng': 1, 'lat': 1,
updateDict = {'city': doc['city'], 'region': doc['region'], 'category': doc['cat'], 'name': doc['name'], 'status': doc['status']} # 'status': 1}) \
if 'estate_type2' in doc: # .batch_size(10000):
updateDict['estate_type'] = doc['estate_type2'] # updateDict = {'city': doc['city'], 'region': doc['region'], 'category': doc['cat'], 'name': doc['name'],
# 'status': doc['status']}
id_list.append({'_id': doc['_id']}) # if 'estate_type2' in doc:
if 'block' in doc: # updateDict['estate_type'] = doc['estate_type2']
updateDict['block'] = doc['block'] #
if 'address' in doc: # id_list.append({'_id': doc['_id']})
updateDict['address'] = doc['address'] # if 'block' in doc:
if 'township' in doc: # updateDict['block'] = doc['block']
updateDict['township'] = doc['township'] # if 'address' in doc:
if 'alias' in doc: # updateDict['address'] = doc['address']
updateDict['alias'] = doc['alias'] # if 'township' in doc:
if 'addr_alias' in doc: # updateDict['township'] = doc['township']
updateDict['addr_alias'] = doc['addr_alias'] # if 'alias' in doc:
if 'lng' in doc and doc['lng'] is not None and str(doc['lng']) != 'nan': # updateDict['alias'] = doc['alias']
updateDict['location'] = {'lon': doc['lng'] / 10**10, 'lat': doc['lat'] / 10**10} # if 'addr_alias' in doc:
updateDict['update_date'] = update_date # updateDict['addr_alias'] = doc['addr_alias']
action = UpdateOne({'_id': doc['_id']}, {'$set': updateDict}, upsert=True) # if 'lng' in doc and doc['lng'] is not None and str(doc['lng']) != 'nan':
# updateDict['location'] = {'lon': doc['lng'] / 10 ** 10, 'lat': doc['lat'] / 10 ** 10}
actions.append(action) # updateDict['update_date'] = update_date
if len(actions) == 10000: # action = UpdateOne({'_id': doc['_id']}, {'$set': updateDict}, upsert=True)
seaweed_didata_conn.bulk_write(actions) #
actions.clear() # actions.append(action)
# if len(actions) == 10000:
didata_id_df = pd.DataFrame(list(seaweed_didata_conn.find({'city': city, 'category': type}, {'_id': 1}).batch_size(10000))) # seaweed_didata_conn.bulk_write(actions)
if len(didata_id_df) > 0: # actions.clear()
didata_id_df = didata_id_df[['_id']] #
id_df = pd.DataFrame(id_list) # didata_id_df = pd.DataFrame(
deleted_seaweed_df = pd.concat([didata_id_df, id_df, id_df]).drop_duplicates(keep=False) # list(seaweed_didata_conn.find({'city': city, 'category': type}, {'_id': 1}).batch_size(10000)))
# if len(didata_id_df) > 0:
for deleted_seaweed_index, deleted_seaweed_item in deleted_seaweed_df.iterrows(): # didata_id_df = didata_id_df[['_id']]
action = UpdateOne({'_id': ObjectId(deleted_seaweed_item['_id'])}, {'$set': {'status': 1, 'update_date': update_date}}, upsert=False) # id_df = pd.DataFrame(id_list)
actions.append(action) # deleted_seaweed_df = pd.concat([didata_id_df, id_df, id_df]).drop_duplicates(keep=False)
if len(actions) == 10000: #
seaweed_didata_conn.bulk_write(actions) # for deleted_seaweed_index, deleted_seaweed_item in deleted_seaweed_df.iterrows():
actions.clear() # action = UpdateOne({'_id': ObjectId(deleted_seaweed_item['_id'])},
# {'$set': {'status': 1, 'update_date': update_date}}, upsert=False)
if len(actions) > 0: # actions.append(action)
seaweed_didata_conn.bulk_write(actions) # if len(actions) == 10000:
actions.clear() # seaweed_didata_conn.bulk_write(actions)
log.info(city + "-" + type + "-同步完成") # actions.clear()
#
if type == 'all': # if len(actions) > 0:
sync_district_office("district") # seaweed_didata_conn.bulk_write(actions)
sync_district_office("office") # actions.clear()
else: # log.info(city + "-" + type + "-同步完成")
sync_district_office(type) #
#
# if type == 'all':
# sync_district_office("district")
# sync_district_office("office")
# else:
# sync_district_office(type)
+161
View File
@@ -0,0 +1,161 @@
from goojia_utils import MAIN_MONGO_CLIENT, dateutils
from goojia_utils import areautils
from datetime import datetime, date
import pandas as pd
from dateutil.relativedelta import relativedelta
import logging
deal_price_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'deal_price')
district_sell_stat_conn = MAIN_MONGO_CLIENT.get_collection('fangjia_stat', 'district_sell_stat')
evaluate_ratio_conn = MAIN_MONGO_CLIENT.get_collection('business', 'evaluate_ratio')
m_date = dateutils.iso_format()
datetimeNow = date.today()
datetimeTwoYearAgo = datetimeNow - relativedelta(years=2)
ago_year = datetimeTwoYearAgo.year
ago_month = datetimeTwoYearAgo.month
ago_day = datetimeTwoYearAgo.day
year = datetimeNow.year
month = datetimeNow.month
day = datetimeNow.day
def sync(city):
# 查询2年内成交价格
dear_price_r = deal_price_conn.find({
'city': city,
'trade_date':
{
"$gte": datetime(ago_year, ago_month, ago_day, 0, 0),
"$lt": datetime(year, month, day, 0, 0)
}
}, {'city': 1, 'region': 1, 'district_name': 1, 'avg_price': 1, 'area': 1, 'trade_date': 1, 'match_id': 1,
'type': 1}).batch_size(10000)
df1 = pd.DataFrame(list(dear_price_r))
if df1.empty:
return
if 'type' in df1:
df1 = df1[df1.type != "车位"]
df1 = df1[df1.area > 20]
df1 = df1[df1.avg_price < 500000]
df1 = df1[~df1.match_id.isna()]
if df1.empty:
return
# 去重
df1.drop_duplicates(subset=['city', 'region', 'district_name', 'avg_price', 'area', 'trade_date', 'match_id'],
keep='first', inplace=True)
df1.reset_index(drop=True, inplace=True)
# 过滤数据(同小区成交价与小区成交均价偏差较大)
df1['中位值'] = df1.groupby(by=['city', 'region', 'district_name'])['avg_price'].transform("median")
df1['ratio'] = round((df1['avg_price'] - df1['中位值']) / df1['中位值'], 2)
df1 = df1.drop(df1.index[df1['ratio'] > 0.25])
df1 = df1.drop(df1.index[df1['ratio'] < -0.25])
# 从日期中提取年月,进行拼接
df1['Time'] = pd.to_datetime(df1['trade_date'])
df1['Year'] = df1['Time'].dt.strftime('%Y').map(str)
df1['Month'] = df1['Time'].dt.strftime('%m').map(str)
df1['date'] = df1['Year'] + df1['Month']
convert_dict = {'date': int}
df1 = df1.astype(convert_dict)
year_month = int(dateutils.format_date(datetimeNow, fm='%Y%m'))
two_year_ago_year_month = int(dateutils.format_date(datetimeTwoYearAgo, fm='%Y%m'))
# 查询2年内挂牌价格
district_stat_r = district_sell_stat_conn.find({
'city': city,
'date':
{
"$gte": two_year_ago_year_month,
"$lt": year_month
}
}, {'city': 1, 'region': 1, 'district_name': 1, 'avg_price': 1, 'date': 1})
logging.info(f"2C价格查询范围{two_year_ago_year_month}-{year_month}")
df2 = pd.DataFrame(list(district_stat_r))
# 过滤数据(同小区挂牌价与小区挂牌均价偏差较大)
df2['中位值'] = df2.groupby(by=['city', 'region', 'district_name'])['avg_price'].transform("median")
df2['ratio'] = round((df2['avg_price'] - df2['中位值']) / df2['中位值'], 2)
df2 = df2.drop(df2.index[df2['ratio'] > 0.25])
df2 = df2.drop(df2.index[df2['ratio'] < -0.25])
# 合并表
df1_2 = pd.merge(df1, df2, how='left', on=['city', 'region', 'district_name', 'date'])
# 过滤数据(成交价与挂牌价偏差较大)
df1_2['ratio'] = round((df1_2['avg_price_x'] - df1_2['avg_price_y']) / df1_2['avg_price_y'], 2)
df1_2 = df1_2.drop(df1_2.index[df1_2['ratio'] > 0.25])
df1_2 = df1_2.drop(df1_2.index[df1_2['ratio'] < -0.25])
df1_2 = df1_2.dropna(axis='index', how='all', subset=['ratio'])
# 城市挂牌价与成交价计算价格系数
deal_price_city_avgprice = df1_2.groupby(by=['city'])['avg_price_x'].mean().reset_index()
district_stat_city_avgprice = df1_2.groupby(by=['city'])['avg_price_y'].mean().reset_index()
df = pd.merge(deal_price_city_avgprice, district_stat_city_avgprice, how='inner', on=['city'])
df['ratio'] = round((df['avg_price_x'] - df['avg_price_y']) / df['avg_price_y'], 2)
# df.to_excel('data/city_ratio.xlsx', index=False)
for index in df.index:
city = df['city'].get(index)
ratio = df['ratio'].get(index)
if ratio > 0.25 or ratio < -0.25:
continue
else:
query = {"city": city, "category": "city"}
update = {"$set": {"ratio": ratio, "m_date": m_date}}
evaluate_ratio_conn.update_one(query, update, upsert=True)
# 区域挂牌价与成交价计算价格系数
deal_price_region_avgprice = df1_2.groupby(by=['city', 'region'])['avg_price_x'].mean().reset_index()
district_stat_region_avgprice = df1_2.groupby(by=['city', 'region'])['avg_price_y'].mean().reset_index()
df = pd.merge(deal_price_region_avgprice, district_stat_region_avgprice, how='inner', on=['city', 'region'])
df['ratio'] = round((df['avg_price_x'] - df['avg_price_y']) / df['avg_price_y'], 2)
for index in df.index:
city = df['city'].get(index)
ratio = df['ratio'].get(index)
region = df['region'].get(index)
if ratio > 0.25 or ratio < -0.25:
continue
else:
query = {"city": city, "region": region, "category": "region"}
update = {"$set": {"ratio": ratio, "region": region, "m_date": m_date}}
evaluate_ratio_conn.update_one(query, update, upsert=True)
# 小区挂牌价与成交价计算价格系数
deal_price_district_avgprice = df1_2.groupby(by=['city', 'region', 'district_name'])[
'avg_price_x'].mean().reset_index()
district_stat_district_avgprice = df1_2.groupby(by=['city', 'region', 'district_name'])[
'avg_price_y'].mean().reset_index()
df = pd.merge(deal_price_district_avgprice, district_stat_district_avgprice, how='inner',
on=['city', 'region', 'district_name'])
df['ratio'] = round((df['avg_price_x'] - df['avg_price_y']) / df['avg_price_y'], 2)
df = df[['city', 'region', 'district_name', 'ratio']]
for index in df.index:
city = df['city'].get(index)
ratio = df['ratio'].get(index)
region = df['region'].get(index)
district_name = df['district_name'].get(index)
if ratio > 0 or ratio < -0.25:
continue
else:
query = {"city": city, "region": region, "district_name": district_name, "category": "district"}
update = {
"$set": {"ratio": ratio, "m_date": m_date}}
evaluate_ratio_conn.update_one(query, update, upsert=True)
if __name__ == "__main__":
for city in areautils.get_all_cities():
logging.info(f"开始计算{city}市估值系数")
sync(city)
logging.info(f"{city}市估值系数计算结束")
+24
View File
@@ -0,0 +1,24 @@
import logging
from goojia_utils import ElasticSearchClient, dingutils
log = logging.getLogger(__name__)
es206_client = ElasticSearchClient(host='192.168.0.206')
from datetime import datetime, timedelta
# 获取当前时间
now = datetime.now()
# 计算当前时间以前2分钟的时间
two_minutes_ago = now - timedelta(minutes=2)
# 将时间转换为时间戳(以毫秒为单位)
# 注意:timestamp() 方法返回的是秒,所以我们需要乘以1000来得到毫秒
timestamp_milliseconds = int(two_minutes_ago.timestamp() * 1000)
count = 0
for doc in es206_client.scan_by_sql("select code from fangjia_log where code >= 500 and creationDate > " + str(timestamp_milliseconds)):
count += 1
if count > 2:
print('【OPEN日志监控】最近两分钟超过两条错误,请检查日志!')
dingutils.send_text('【OPEN日志监控】最近两分钟超过两条错误,请检查OPEN日志!', 'ab4231000995a8c90d8d1e91dfd6930c3896dd1c3bbf1f0af19e19004cf6c98c')
break
+151
View File
@@ -0,0 +1,151 @@
import logging
import sys
from goojia_utils import ElasticSearchClient, commutils
import pandas as pd
from elasticsearch import helpers
import transbigdata as tbd
log = logging.getLogger(__name__)
city_name = sys.argv[1] if len(sys.argv) > 1 else None
if pd.notna(city_name):
log.info("同步城市:" + city_name)
es206_client = ElasticSearchClient(host='192.168.0.206')
# 中国地理位置边界[lon1,lat1,lon2,lat2] (WGS84坐标系) 其中,lon1,lat1是左下角坐标,lon2,lat2是右上角坐标,
china_bounds = [73.66, 3.86, 135.05, 53.55]
# 栅格大小(米)
accuracy = 1000
# 生成栅格参数
params_rect = tbd.area_to_params(china_bounds, accuracy=accuracy)
# def find_comm_by_grid(city, geo_shape):
# comm_info_df = es206_client.scan_to_data_frame('community_summary', {
# "_source": ["id", "city", "region", "name", "estate_type"],
# "query": {
# "bool": {
# "filter": [
# {
# "term": {
# "city": {
# "value": city
# }
# }
# },
# {
# "geo_polygon": {
# "location": {
# "points": geo_shape
# }
# }
# }
# ]
# }
# }
# })
# return comm_info_df
estate_type_list = ["普通住宅", "别墅", "商住"]
def save_to_community_summary(comm_df):
actions = list()
count = 0
for index, item in comm_df.iterrows():
if pd.isna(item['grid_id']):
continue
doc_info = {'grid_id': item['grid_id']}
action = {
"_op_type": "update",
"_index": "community_summary",
"_id": item['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新:" + str(count))
actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
actions.clear()
def sync(city):
sql = f"select * from grid "
if pd.notna(city):
sql += f" where city='{city}' "
origin_grid_df = es206_client.scan_by_sql_to_data_frame(sql)
cities = origin_grid_df['city'].unique()
for curr_city in cities:
grid_df = origin_grid_df[origin_grid_df['city'] == curr_city].copy()
avg_price_df = commutils.get_avg_price(curr_city)
if len(avg_price_df) <= 0:
logging.info(curr_city + "无价格信息")
continue
avg_price_df = avg_price_df[['city', 'region', 'district_name', 'avg_price']]
avg_price_df = avg_price_df.rename(columns={'district_name': 'name'})
comm_df = es206_client.scan_by_sql_to_data_frame(f"select id, city, region, name, estate_type, location from community_summary where city='{curr_city}' ")
def deal_lng_lat(x):
location = x['location']
if pd.notna(location):
lon_lat_col = tbd.GPS_to_grid(location['lon'], location['lat'], params_rect)
x['grid_id'] = str(lon_lat_col[0]) + "_" + str(lon_lat_col[1])
return x
comm_df = comm_df.apply(lambda x: deal_lng_lat(x), axis=1)
# 格子信息写入community_summary
save_to_community_summary(comm_df)
# 求解格子均价
actions = list()
count = 0
log.info(f"开始计算{curr_city}格子价格")
for index, item in grid_df.iterrows():
comm_info_df = comm_df[comm_df['grid_id'] == item['_id']]
if len(comm_info_df) <= 0:
continue
grid_info = dict()
price_list = list()
for estate_type in estate_type_list:
# 计算不同物业类型的小区均价
comm_info_estate_type_df = comm_info_df[comm_info_df['estate_type'] == estate_type].copy()
if len(comm_info_estate_type_df) > 0:
comm_info_estate_type_df = pd.merge(comm_info_estate_type_df, avg_price_df, how='left', on=['city', 'region', 'name'])
comm_info_estate_type_df = comm_info_estate_type_df[comm_info_estate_type_df['avg_price'].notna()]
if len(comm_info_estate_type_df) > 0:
price_info = dict()
price_info['type'] = 'district'
price_info['secondary_type'] = estate_type
price_info['price'] = int(comm_info_estate_type_df['avg_price'].mean())
price_list.append(price_info)
if len(price_list) > 0:
grid_info['price'] = price_list
action = {
"_op_type": "update",
"_index": "grid",
"_id": item['_id'],
"doc": grid_info,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新格子数:" + str(count))
actions.clear()
if len(actions) > 0:
helpers.bulk(es206_client.elastic_client, actions)
count += len(actions)
log.info("更新格子数:" + str(count))
actions.clear()
log.info(f"完成计算{curr_city}格子价格")
sync(city_name)
+3
View File
@@ -0,0 +1,3 @@
from goojia_utils import KafkaClient
KAFKA_CLIENT = KafkaClient()
KAFKA_CLIENT.send_message("email-data-topic","sendEmail")
+3
View File
@@ -0,0 +1,3 @@
from goojia_utils import KafkaClient
KAFKA_CLIENT = KafkaClient()
KAFKA_CLIENT.send_message("sms-data-topic","sendEndTimeSms")
@@ -1,13 +1,13 @@
from airflow.operators.bash import BashOperator from airflow.providers.standard.operators.bash import BashOperator
import pendulum import pendulum
from airflow import DAG from airflow import DAG
with DAG( with DAG(
"amap_key_generate_dag", dag_id="amap_key_generate_dag",
# 每天运行 # 每天运行
schedule_interval="0 2 * * *", schedule="0/1 14 * * *",
start_date=pendulum.datetime(2022, 7, 18, tz="Asia/Shanghai"), start_date=pendulum.datetime(2023, 3, 18, tz="Asia/Shanghai"),
catchup=True, catchup=True
) as dag: ) as dag:
t1 = BashOperator( t1 = BashOperator(
task_id='amap_key_generate', task_id='amap_key_generate',
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="apiuser_notice_dag",
# 每小时运行
schedule="0 * * * * ",
start_date=pendulum.datetime(2023, 7, 10, tz="Asia/Shanghai"),
catchup=True,
description="API接口监控提醒"
) as dag:
t1 = BashOperator(
task_id='apiuser_notice',
bash_command='python /opt/airflow/dags/repo/dags/business/business/apiuser_notice.py',
)
t1
@@ -1,11 +1,11 @@
from airflow.operators.bash import BashOperator from airflow.providers.standard.operators.bash import BashOperator
import pendulum import pendulum
from airflow import DAG from airflow import DAG
with DAG( with DAG(
"change_new_to_old_seaweed_dag", dag_id="change_new_to_old_seaweed_dag",
# 每月1日1点运行 # 每月1日1点运行
schedule_interval="0 1 1 * *", schedule="0 1 1 * *",
start_date=pendulum.datetime(2023, 2, 10, tz="Asia/Shanghai"), start_date=pendulum.datetime(2023, 2, 10, tz="Asia/Shanghai"),
catchup=True, catchup=True,
description="新房变二手房定时脚本" description="新房变二手房定时脚本"
+59
View File
@@ -0,0 +1,59 @@
# from airflow.providers.docker.operators.docker import DockerOperator
# from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
# import pendulum
# from airflow import DAG
# from docker.types import Mount
# from kubernetes.client import models as k8s
#
# with DAG(
# dag_id="comm_add_dag",
# # 每月20日1点整分运行
# schedule="0 1 20 * *",
# start_date=pendulum.datetime(2023, 5, 8, tz="Asia/Shanghai"),
# catchup=True,
# description="小区自动化入库"
# ) as dag:
# # k = KubernetesPodOperator(
# # name="comm-add",
# # image="192.168.0.19:9012/comm-add:1.0.1",
# # cmds=["bash", "-cx"],
# # arguments=["echo", "10"],
# # labels={"foo": "bar"},
# # task_id="comm_add",
# # do_xcom_push=True,
# # )
# # k.dry_run()
#
# production_task = KubernetesPodOperator(namespace='airflow-plus',
# # image="my-production-job:release-1.0.1", <-- old release
# image="192.168.0.19:9012/comm-add:1.0.1",
# image_pull_policy="Always",
# image_pull_secrets=[k8s.V1LocalObjectReference("nexus-registry-secret")],
# cmds=["python", "main.py"],
# arguments=["--city", '{{dag_run.conf["city"] if "city" in dag_run.conf else "all"}}'],
# name="comm_add",
# task_id="comm_add",
# get_logs=True
# )
#
# production_task
# # t1 = DockerOperator(
# # # 刚才定义的conn id
# # docker_conn_id="docker_repo",
# # # WebServer中点击Admin -> Providers -> Docker对应的Version,如果是auto,可能会报错
# # api_version="2.2.0",
# # auto_remove=True,
# # docker_url='unix://var/run/docker.sock',
# # network_mode='bridge',
# # image='192.168.0.19:9012/comm-add:1.0.1',
# # # 临沂为我们的传参,python代码中, 镜像中部分代码
# # # @ click.command()
# # # @ click.option('--city', default="all", help='要计算的城市')
# # #command='python main.py --city {{dag_run.conf["city"] if "city" in dag_run.conf}} else all',
# # command='echo "this is a test message shown from within the container',
# # mounts=[
# # Mount(source="/var/run/docker.sock", target="/var/run/docker.sock", type="bind")
# # ],
# # task_id='comm_add'
# # )
# # t1
@@ -1,15 +1,19 @@
from airflow.operators.bash import BashOperator from airflow.providers.standard.operators.bash import BashOperator
import pendulum import pendulum
from airflow import DAG from airflow import DAG
with DAG( with DAG(
"community_summary_dag", dag_id="community_summary_dag",
# 每月2日1点整分运行 # 每月2日1点整分运行
schedule_interval="0 1 2 * *", schedule="0 1 2 * *",
start_date=pendulum.datetime(2022, 10, 12, tz="Asia/Shanghai"), start_date=pendulum.datetime(2026, 8, 16, tz="Asia/Shanghai"),
catchup=True, catchup=True,
description="地址输出信息分类数据定时更新" description="地址输出信息分类数据定时更新"
) as dag: ) as dag:
del_comm = BashOperator(
task_id='del_comm',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/del_comm.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
comm = BashOperator( comm = BashOperator(
task_id='comm', task_id='comm',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/comm.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/comm.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
@@ -58,6 +62,10 @@ with DAG(
task_id='relative_price_ranking', task_id='relative_price_ranking',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/relative_price_ranking.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/relative_price_ranking.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
) )
# relative_price_ranking_fix = BashOperator(
# task_id='relative_price_ranking_fix',
# bash_command='python /opt/airflow/dags/repo/dags/business/business/community/relative_price_ranking_fix.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
# )
estate_charge_complete = BashOperator( estate_charge_complete = BashOperator(
task_id='estate_charge_complete', task_id='estate_charge_complete',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/estate_charge_complete.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/estate_charge_complete.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
@@ -70,10 +78,30 @@ with DAG(
task_id='consumption', task_id='consumption',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/consumption.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/consumption.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
) )
city_level_and_price_rating = BashOperator(
task_id='city_level_and_price_rating',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/city_level_and_price_rating.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
credit = BashOperator(
task_id='credit',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/credit.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
person_rating = BashOperator(
task_id='person_rating',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/person_rating.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
price_rating = BashOperator( price_rating = BashOperator(
task_id='price_rating', task_id='price_rating',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/price_rating.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/price_rating.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
) )
consumption_stat = BashOperator(
task_id='consumption_stat',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/consumption_stat.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
percentile = BashOperator(
task_id='percentile',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/percentile.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
education = BashOperator( education = BashOperator(
task_id='education', task_id='education',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/education.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/education.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
@@ -90,5 +118,12 @@ with DAG(
task_id='other', task_id='other',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/other.py {{dag_run.conf["city"] if "city" in dag_run.conf}}', bash_command='python /opt/airflow/dags/repo/dags/business/business/community/other.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
) )
comm >> price >> avg_cost >> total_price >> total_price_complete >> relative_price_ranking >> estate_charge_complete >> estate_charge_ranking \ # 统一输出为省市区全拼
>> consumption >> price_rating >> sell >> trade >> traffic >> business_district >> medical >> life >> education >> education2 >> extra >> other full_name = BashOperator(
task_id='full_name',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/full_name.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
# 人均消费并行,值为空
del_comm >> comm >> price >> [avg_cost, total_price, relative_price_ranking, estate_charge_complete] >> total_price_complete >> consumption >> city_level_and_price_rating >> credit >> [estate_charge_ranking, price_rating, person_rating, percentile] >> \
consumption_stat >> sell >> trade >> [traffic, business_district, medical, life, education] >> education2 >> extra >> other \
>> full_name
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
# with DAG(
# dag_id="didata_sync_dag",
# # 每月28日3点整分运行
# schedule="0 3 28 * *",
# start_date=pendulum.datetime(2023, 3, 17, tz="Asia/Shanghai"),
# catchup=True,
# description="地数相关数据同步type: district或office, 不填默认同步全部"
# ) as dag:
# t1 = BashOperator(
# task_id='didata_sync_dag',
# bash_command='python /opt/airflow/dags/repo/dags/business/business/didata_sync.py {{dag_run.conf["type"] if "type" in dag_run.conf}}',
# )
# t1
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="evaluate_ratio_dag",
# 每月1日0点运行
schedule="0 0 1 * *",
start_date=pendulum.datetime(2023, 7, 5, tz="Asia/Shanghai"),
catchup=True,
description="挂牌价与成交价计算价格系数"
) as dag:
t1 = BashOperator(
task_id='evaluate_ratio',
bash_command='python /opt/airflow/dags/repo/dags/business/business/evaluate_ratio.py',
)
t1
+19
View File
@@ -0,0 +1,19 @@
from datetime import timedelta
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="fangjia_log_dag",
# 每2分钟监控
schedule=timedelta(minutes=2),
start_date=pendulum.datetime(2024, 8, 14, tz="Asia/Shanghai"),
catchup=True,
description="OPEN日志监控"
) as dag:
t1 = BashOperator(
task_id='fangjia_log',
bash_command='python /opt/airflow/dags/repo/dags/business/business/fangjia_log.py',
)
t1
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="grid_price_dag",
# 每周二运行
schedule="0 3 * * 2",
start_date=pendulum.datetime(2023, 5, 9, tz="Asia/Shanghai"),
catchup=True,
description="全国格子价格计算"
) as dag:
t1 = BashOperator(
task_id='grid_price',
bash_command='python /opt/airflow/dags/repo/dags/business/business/grid_price.py {{dag_run.conf["city"] if "city" in dag_run.conf}}',
)
t1
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="send_email",
# 每月1日8点运行
schedule="0 8 1 * *",
start_date=pendulum.datetime(2024, 8, 30, tz="Asia/Shanghai"),
catchup=True,
description="API用户使用及明细的邮件发送"
) as dag:
t1 = BashOperator(
task_id='send_email',
bash_command='python /opt/airflow/dags/repo/dags/business/business/send_email.py',
)
t1
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="send_sms",
# 每月1日8点运行
schedule="0 8 * * *",
start_date=pendulum.datetime(2024, 9, 25, tz="Asia/Shanghai"),
catchup=True,
description="API用户使用及明细的邮件发送"
) as dag:
t1 = BashOperator(
task_id='send_sms',
bash_command='python /opt/airflow/dags/repo/dags/business/business/send_sms.py',
)
t1
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="sync_es_price_to_mongo_dag",
# 每月3日凌晨2点整执行(在大部分上游爬取和ES清洗完毕后进行数据同步归档)
schedule="0 2 3 * *",
start_date=pendulum.datetime(2026, 8, 16, tz="Asia/Shanghai"),
catchup=False,
description="每月定时把 ES 206 上的 community 均价数据同步到 85 Mongo 的 average_statistics 表"
) as dag:
sync_es_price_to_mongo = BashOperator(
task_id='sync_es_price_to_mongo',
bash_command='python /opt/airflow/dags/repo/dags/business/business/community/sync_es_price_to_mongo.py {{dag_run.conf["month"] if "month" in dag_run.conf else ""}}',
)
-17
View File
@@ -1,17 +0,0 @@
from airflow.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
"didata_sync_dag",
# 每月28日3点整分运行
schedule_interval="0 3 28 * *",
start_date=pendulum.datetime(2022, 8, 22, tz="Asia/Shanghai"),
catchup=True,
description="地数相关数据同步type: district或office, 不填默认同步全部"
) as dag:
t1 = BashOperator(
task_id='didata_sync_dag',
bash_command='python /opt/airflow/dags/repo/dags/business/business/didata_sync.py {{dag_run.conf["type"] if "type" in dag_run.conf}}',
)
t1
View File
View File
+87
View File
@@ -0,0 +1,87 @@
"""
第一步: 将百度经纬度转为高德经纬度(只需跑批一次即可) TODO
"""
import os
# 1. 尝试环境变量兼容
os.environ["ELASTIC_CLIENT_APIVERSIONING"] = "0"
from goojia_utils import DIDATA_DEV_ELASTIC_CLIENT, threads, geohashutils, areautils
import pandas as pd
from elasticsearch import helpers
import logging
log = logging.getLogger(__name__)
# ================== 🛡️ 安全运行入口 ==================
if __name__ == "__main__":
flag = False
for province in areautils.get_all_province():
print(f"开始处理{province}数据")
# 百度经纬度转高德经纬度
df = DIDATA_DEV_ELASTIC_CLIENT.scan_to_data_frame(index='didata_enterprise', query={
"query": {
"bool": {
"must": [
{
"term": {
"province": {
"value": "" + province + ""
}
}
}
],
"must_not": [
{
"exists": {
"field": "gaode_location"
}
}
]
}
}
})
if len(df) == 0:
continue
def deal_location(item):
if pd.isna(item['location']):
return item
gd_location = geohashutils.bd_to_gd(item['location']['lon'], item['location']['lat'])
item['gaode_location'] = {'lon': gd_location[0], 'lat': gd_location[1]}
return item
my_df = threads.process_dataframe_multithread(df, deal_location, num_threads=100)
actions = list()
count = 0
for index, item in my_df[my_df['gaode_location'].notna()].iterrows():
doc_info = {'gaode_location': item['gaode_location']}
action = {
"_op_type": "update",
"_index": "didata_enterprise",
"_id": item['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
if len(actions) == 5000:
for success, info in helpers.parallel_bulk(
# 此时这里的底层 elastic_client 将自动套用强行重置后的 JSON 请求头
DIDATA_DEV_ELASTIC_CLIENT.elastic_client,
actions,
thread_count=5, # 并行线程数
chunk_size=1000 # 批量大小
):
if not success:
print(f"索引失败: {info}")
count += len(actions)
print("更新:" + str(count))
actions.clear()
if len(actions) > 0:
helpers.bulk(DIDATA_DEV_ELASTIC_CLIENT.elastic_client, actions)
count += len(actions)
print("更新:" + str(count))
actions.clear()
+108
View File
@@ -0,0 +1,108 @@
"""
通过居委围栏更新隶属关系
"""
from goojia_utils import DIDATA_DEV_ELASTIC_CLIENT, areautils
import pandas as pd
from elasticsearch import helpers
import logging
# 配置日志
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
def sync(province):
logger.info(f"开始同步{province}企业数据的居委村委数据")
shape_df = DIDATA_DEV_ELASTIC_CLIENT.scan_by_sql_to_data_frame(
f"select city_name, area_name, street_name, name, shape from community where province_name='{province}'")
if len(shape_df) == 0 or 'shape' not in shape_df.columns or len(shape_df[shape_df['shape'].notna()]) == 0:
logger.info(f"{province}无围栏数据")
return
logger.info(f"已加载{len(shape_df)}条围栏")
# 先临时禁用索引刷新
DIDATA_DEV_ELASTIC_CLIENT.elastic_client.indices.put_settings(
index="didata_company",
body={"index": {"refresh_interval": "-1"}}
)
ids = DIDATA_DEV_ELASTIC_CLIENT.scan_by_sql_to_data_frame(f"select company_name, committee from didata_company where province='{province}'")
#ids = pd.DataFrame({})
if len(ids) > 0:
company_ids = ids['_id'].to_list()
else:
company_ids = list()
actions = list()
count = 0
errors = list()
for index, item in shape_df[shape_df['shape'].notna()].iterrows():
logger.info(f"开始查找{item['city_name'], item['area_name'], item['name']}下企业数据")
ret = DIDATA_DEV_ELASTIC_CLIENT.fetch_all_geo_points('didata_enterprise', item['shape']['coordinates'][0], field='gaode_location')
com_count = 0
for info in ret:
com_count += 1
source = info['_source']
if 'district' in source and (source['district'] != item['area_name']):
errors.append(source)
continue
if ('registered_address' not in source) or ('company_name' not in source) or ('district' not in source) or ('city' not in source):
continue
doc_info = {'province': source['province'], 'city': source['city'], 'district': source['district'],
'company_name': source['company_name'], 'registered_address': source['registered_address'],
'location': source['gaode_location'], 'township': item['street_name'],
'committee': item['name']}
if info['_id'] not in company_ids:
action = {
"_op_type": "index",
"_index": "didata_company",
"_id": info['_id'],
"_source": doc_info,
"_retry_on_conflict": 5
}
else:
action = {
"_op_type": "update",
"_index": "didata_company",
"_id": info['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
#pd.DataFrame(errors)[['registered_address', 'company_name', 'province', 'city', 'district', 'gaode_location']].to_csv(f'{province}经纬度错误地址.csv', index=False)
logger.info(f"已查找到该围栏下{com_count}个企业数据")
for success, info in helpers.parallel_bulk(
DIDATA_DEV_ELASTIC_CLIENT.elastic_client,
actions,
thread_count=5, # 并行线程数
refresh=False,
chunk_size=1000 # 批量大小
):
if not success:
logger.info(f"索引失败: {info}")
count += len(actions)
logger.info("更新:" + str(count))
actions.clear()
if len(actions) > 0:
for success, info in helpers.parallel_bulk(
DIDATA_DEV_ELASTIC_CLIENT.elastic_client,
actions,
thread_count=5, # 并行线程数
refresh=False,
chunk_size=1000 # 批量大小
):
if not success:
logger.error(f"索引失败: {info}")
count += len(actions)
logger.info("更新:" + str(count))
actions.clear()
#pd.DataFrame(errors)[['registered_address', 'company_name', 'province', 'city', 'district', 'gaode_location']].to_csv(f'{province}经纬度错误地址.csv', index=False)
logger.info(f"完成同步{province}企业数据的居委村委数据")
# 恢复索引刷新
DIDATA_DEV_ELASTIC_CLIENT.elastic_client.indices.put_settings(
index="didata_company",
body={"index": {"refresh_interval": "5s"}}
)
if __name__ == '__main__':
flag = False
# for province in areautils.get_all_province():
# print(province)
for province in areautils.get_all_province():
sync(province)
+79
View File
@@ -0,0 +1,79 @@
from goojia_utils import DIDATA_DEV_ELASTIC_CLIENT, areautils
import pandas as pd
from elasticsearch import helpers
import numpy as np
import re
import logging
# 配置日志
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
"""
通过开发区围栏更新企业开发区 TODO
"""
def sync(province):
logger.info(f"开始同步{province}企业数据的开发区数据")
# 先临时禁用索引刷新
DIDATA_DEV_ELASTIC_CLIENT.elastic_client.indices.put_settings(
index="didata_company",
body={"index": {"refresh_interval": "-1"}}
)
shape_df = DIDATA_DEV_ELASTIC_CLIENT.scan_by_sql_to_data_frame(f"select name, shape from development_zone where province='{province}'")
if len(shape_df) == 0 or 'shape' not in shape_df.columns or len(shape_df[shape_df['shape'].notna()]) == 0:
logger.info(f"{province}无围栏数据")
return
actions = list()
count = 0
for index, item in shape_df[shape_df['shape'].notna()].iterrows():
logger.info(f"开始查找{item['name']}下企业数据")
ret = DIDATA_DEV_ELASTIC_CLIENT.fetch_all_geo_points('didata_company', item['shape']['coordinates'][0],
field='location')
com_count = 0
for info in ret:
com_count += 1
doc_info = {'dev': item['name']}
action = {
"_op_type": "update",
"_index": "didata_company",
"_id": info['_id'],
"doc": doc_info,
"_retry_on_conflict": 5
}
actions.append(action)
logger.info(f"已查找到该围栏下{com_count}个企业数据")
if len(actions) > 0:
for success, info in helpers.parallel_bulk(
DIDATA_DEV_ELASTIC_CLIENT.elastic_client,
actions,
thread_count=5, # 并行线程数
chunk_size=10000 # 批量大小
):
if not success:
print(f"索引失败: {info}")
count += len(actions)
print("更新:" + str(count))
actions.clear()
if len(actions) > 0:
for success, info in helpers.parallel_bulk(
DIDATA_DEV_ELASTIC_CLIENT.elastic_client,
actions,
thread_count=5, # 并行线程数
chunk_size=10000 # 批量大小
):
if not success:
print(f"索引失败: {info}")
count += len(actions)
print("更新:" + str(count))
actions.clear()
logger.info(f"完成同步{province}企业数据的开发区数据")
# 恢复索引刷新
DIDATA_DEV_ELASTIC_CLIENT.elastic_client.indices.put_settings(
index="didata_company",
body={"index": {"refresh_interval": "5s"}}
)
if __name__ == '__main__':
for province in areautils.get_all_province():
sync(province)
+134
View File
@@ -0,0 +1,134 @@
from goojia_utils import DIDATA_DEV_ELASTIC_CLIENT, areautils
import pandas as pd
from elasticsearch import helpers
import re
import logging
log = logging.getLogger(__name__)
def sync():
"""
对于没有围栏结果的,通过match_phrase匹配出来的开发区一定正确,如果有围栏出来的开发区,则企业经纬度不正确
"""
dev_alias = {'福建永定工业园区': '永定区永定工业园', '福建漳州古雷港经济开发区': '古雷港经济开发区', '福建莆田湄洲湾北岸经济开发区': '湄洲湾北岸经济开发区',
'福建南平工业园区': '南平工业园区', '泉州台商投资区': '泉州市台商投资区', '福建洛江经济开发区': '洛江园区', '福建武平工业园区': '武平工业园',
'福建晋江经济开发区': '晋江市经济开发区', '漳州台商投资区': '龙池开发区', '莆田高新技术产业开发区': '莆田市高新技术',
'莆田高新技术产业开发区': '莆田市涵江区高新', '绥安工业开发区': '绥安工业园', '绥安工业开发区': '黄仓开发区', '绥安工业开发区': '黄仓工业园',
'绥安工业开发区': '旧镇工业园', '绥安工业开发区': '大南坂开发区', '绥安工业开发区': '大南坂工业开发区', '福建龙州工业园区': '龙州工业园',
'长汀经济开发区':'腾飞开发区', '福建福州金山工业园区': '金山投资区', '厦门象屿综合保税区': '厦门现代物流园', '福清融侨经济技术开发区': '融侨经济',
'福建洛江经济开发区': '万安开发区'}
dev_df = DIDATA_DEV_ELASTIC_CLIENT.scan_by_sql_to_data_frame("select province, city, district, name, alias from development_zone")
# 不要去重,有可能存在别名, 如福建云霄常山经济开发区
#dev_df = dev_df[dev_df['name'] == '福建政和经济开发区']
actions = list()
for dev in dev_df.itertuples():
if dev.name in ['中国(福建)自由贸易试验区厦门片区', '中国(福建)自由贸易试验区福州片区', '中国(福建)自由贸易试验区平潭片区']:
# 中国(福建)自由贸易试验区 已包含,防止反复更新
continue
if dev.name in ['政和经济开发区']:
continue
count = 0
names = list()
names.append(dev.name)
if str(dev.alias) != 'nan':
names.extend(dev.alias)
#东侨经济技术开发区 福建省宁德市东侨经济开发区塔山村塔兴路25号
if '经济技术开发区' in dev.name:
names.append(re.sub('经济技术开发区', '经济开发区', dev.name))
# 福州经济技术开发区 福州开发区君竹路83号科技发展中心大楼第四层R472室(自贸试验区内)
names.append(re.sub('经济技术开发区', '开发区', dev.name))
# 长汀经济开发区 长汀县经济开发区工业新区河梁大山体
if '经济开发区' in dev.name:
if pd.notna(dev.city) and pd.notna(dev.district):
district_short = areautils.get_district_short(dev.city, dev.district)
if pd.notna(district_short):
names.append(re.sub(district_short, dev.district, dev.name))
# 漳州高新技术产业开发区 福建省漳州高新区九湖镇木棉村后壁山401-1号
if '高新技术产业开发区' in dev.name:
names.append(re.sub('高新技术产业开发区', '高新区', dev.name))
# 厦门火炬高技术产业开发区 厦门火炬高新区软件园三期溪西山尾路63号1303室-3单元之三
if '高技术产业开发区' in dev.name:
names.append(re.sub('高技术产业开发区', '高新区', dev.name))
#areautils.get_district_display()
# 福建诏安工业园区 福建省漳州市诏安工业园区丹诏大道15号
if pd.notna(dev.city):
province_short = areautils.get_province(dev.city)
if pd.notna(province_short):
new_name = re.sub(province_short, '', dev.name)
if pd.notna(dev.city) and pd.notna(dev.district):
city_short = areautils.get_city_short(dev.city)
district_short = areautils.get_district_short(dev.city, dev.district)
if pd.notna(city_short) and city_short in new_name:
names.append(new_name)
if pd.notna(district_short) and district_short in new_name:
names.append(new_name)
# 福建省漳州市长泰县经济开发区林墩办事处帝君路 福建长泰经济开发区
new_new_name = re.sub(district_short, district_short + '县', new_name)
names.append(new_new_name)
new_new_name = re.sub(district_short, district_short + '区', new_name)
names.append(new_new_name)
for key in dev_alias:
if dev.name == key:
names.append(dev_alias[key])
for name in names:
query = {
"query": {
"bool": {
"must_not": [
{
"term": {
"dev": {
"value": dev.name
}
}
},
{
"term": {
"dev_status": {
"value": 0
}
}
}
],
"must": [
{
"term": {
"province": {
"value": dev.province
}
}
},
{
"match_phrase": {
"registered_address": name
}
}
]
}
}
}
count_data = DIDATA_DEV_ELASTIC_CLIENT.elastic_client.count(index='didata_company',body=query)
if count_data['count'] > 0:
for doc in DIDATA_DEV_ELASTIC_CLIENT.scan(index='didata_company',query=query):
action = {
"_op_type": "update",
"_index": "didata_company",
"_id": doc['_id'],
"doc": {'dev': dev.name, 'dev_status': 0}
}
actions.append(action)
if len(actions) == 5000:
helpers.bulk(DIDATA_DEV_ELASTIC_CLIENT.elastic_client, actions)
count += len(actions)
print(f"已更新 {dev.name} 开发区" + str(count) + "记录")
actions.clear()
if len(actions) > 0:
helpers.bulk(DIDATA_DEV_ELASTIC_CLIENT.elastic_client, actions)
count += len(actions)
print(f"已更新 {dev.name} 开发区" + str(count) + "记录")
actions.clear()
if __name__ == '__main__':
sync()
+26
View File
@@ -0,0 +1,26 @@
from datetime import timedelta
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="didata_company",
schedule=None,
start_date=pendulum.datetime(2025, 6, 5, tz="Asia/Shanghai"),
catchup=True,
description="企业涉农处理"
) as dag:
# t1 = BashOperator(
# task_id='fill_commpany_comm',
# bash_command='python /opt/airflow/dags/repo/dags/company/company/fill_company_comm.py',
# )
# t2 = BashOperator(
# task_id='fill_commpany_dev',
# bash_command='python /opt/airflow/dags/repo/dags/company/company/fill_company_dev.py',
# )
t3 = BashOperator(
task_id='mark_right_dev',
bash_command='python /opt/airflow/dags/repo/dags/company/company/mark_right_dev.py',
)
t3
+73
View File
@@ -0,0 +1,73 @@
CREATE TABLE etl.company (
id Int64,
district String,
ent_name String,
reg_addr String,
unified_code String,
authority String
) ENGINE MySQL('192.168.0.20:3306', 'database', 'td_gov_company_basic', 'root', '12345678')
SETTINGS
connection_wait_timeout=9999999,
connect_timeout=9999999,
read_write_timeout=9999999,
connection_auto_close=false;
CREATE TABLE etl.dwd_company (
district String,
ent_name String,
reg_addr String,
unified_code String,
authority String,
region_code String,
reg_addr1 String,
province_code String,
city_code String,
province_name String,
city_name String,
region_name String,
m_date DateTime('Asia/Shanghai') default now() COMMENT '更新时间'
) ENGINE ReplacingMergeTree(m_date)
PARTITION BY province_code PRIMARY KEY (unified_code) ORDER BY (unified_code);
CREATE TABLE etl.dwm_company (
unified_code String,
province String,
city String,
region String,
name String,
address String,
m_date DateTime('Asia/Shanghai') default now() COMMENT '更新时间'
) ENGINE ReplacingMergeTree(m_date)
PARTITION BY province PRIMARY KEY (unified_code) ORDER BY (unified_code);
INSERT INTO etl.dwm_company(unified_code, province, city, region, name, address) SELECT unified_code, province_name, city_name, region_name, ent_name, reg_addr1 FROM etl.dwd_company WHERE city_name !='' and region_name != '' and reg_addr1 != '';
# 导入company.csv
clickhouse-client --format_csv_delimiter="|" --input_format_with_names_use_header=0 --query="INSERT INTO etl.dwd_company FORMAT CSVWithNames" --host=192.168.0.216 --password=goojia123456 < company.csv
# 导出company.csv
clickhouse-client --password goojia123456 -d etl -q "select * from company FORMAT CSVWithNames" --format_csv_delimiter='|' > company.csv
# 执行sql
clickhouse-client --password goojia123456 -d etl -d dwd_company -m < multi.sql
CREATE TABLE etl.company_local ON CLUSTER fangjia
(
district String,
ent_name String,
reg_addr String,
unified_code String,
authority String
) ENGINE=MergeTree PARTITION BY substring(unified_code, 2, 2) PRIMARY KEY(unified_code) ORDER BY (unified_code)
CREATE TABLE etl.company_shard ON CLUSTER fangjia
(
district String,
ent_name String,
reg_addr String,
unified_code String,
authority String
) ENGINE=Distributed(fangjia, etl, company, hiveHash(substring(unified_code, 2, 2)))
clickhouse-client --format_csv_delimiter="|" --input_format_with_names_use_header=0 --query="INSERT INTO etl.company_shard FORMAT CSVWithNames" --input_format_allow_errors_ratio=0.3 --host=192.168.0.216 --password=goojia123456 < td_gov_company_basic.csv
+30
View File
@@ -0,0 +1,30 @@
CREATE TABLE etl.company_report (
id String,
company_id String,
unified_code String,
year String,
company String,
mail_address String,
m_date DateTime('Asia/Shanghai') default now() COMMENT '更新时间'
) ENGINE ReplacingMergeTree(m_date)
PARTITION BY substring(unified_code, 2, 2) PRIMARY KEY (id) ORDER BY (id);
# 导入数据
clickhouse-client --format_csv_delimiter="@" --input_format_with_names_use_header=0 --query="INSERT INTO etl.company_report FORMAT CSV" --host=192.168.0.216 --password=goojia123456 < part-r-00000
# 关联导入,可能DataGrip客户端超时,就在ClickHouse-Client命令行运行即可
INSERT INTO etl.dwd_company(district, ent_name, reg_addr, unified_code, authority, region_code, reg_addr1, province_code, city_code, province_name, city_name, region_name, mail_address)
SELECT district, ent_name, reg_addr, unified_code, authority, region_code, reg_addr1, province_code, city_code, province_name, city_name, region_name, cr.mail_address
FROM etl.dwd_company c
LEFT JOIN (
SELECT unified_code, argMax(mail_address, year) mail_address, argMax(year, year) new_year FROM etl.company_report GROUP BY unified_code
) cr ON c.unified_code=cr.unified_code
WHERE cr.mail_address!='' and cr.mail_address is not null;
# 合并
optimize table etl.dwd_company final;
# 清洗企业通信地址
ALTER TABLE etl.dwd_company update mail_address1=replaceRegexpAll(mail_address, '^(.{2,}(省|自治区))?(.{2,}市)?(.{2,}(区|县))?', '') WHERE 1=1
INSERT INTO etl.dwm_company(unified_code, province, city, region, name, address) SELECT unified_code, province_name, city_name, region_name, ent_name, reg_addr1 FROM etl.dwd_company WHERE city_name !='' and region_name != '' and reg_addr1 != '';
+27
View File
@@ -0,0 +1,27 @@
create table etl.region_local on cluster fangjia (
province_code String,
fangjia_province String,
province_name String,
city_code String,
city_name String,
fangjia_city String,
region_code String,
fangjia_area String,
region_name String
)
ENGINE MergeTree()
PARTITION BY province_code PRIMARY KEY (region_code) ORDER BY (region_code);
CREATE TABLE etl.region_shard ON CLUSTER fangjia
(
province_code String,
fangjia_province String,
province_name String,
city_code String,
city_name String,
fangjia_city String,
region_code String,
fangjia_area String,
region_name String
) ENGINE=Distributed(fangjia, etl, region_local, hiveHash(province_code))
@@ -0,0 +1,25 @@
create table fangjia_stat.community_coverage_stat
(
city String,
city_level Int32,
community_count Int32,
avg_price_count Int32,
avg_price_coverage Float32,
sqmetre_rent_count Int32,
sqmetre_rent_coverage Float32,
weekend Date,
alias_count Nullable(Int32),
addr_alias_count Nullable(Int32),
one_year_deal Nullable(Int32),
exist_alias_community_count Nullable(Int32),
main_type_coverage Nullable(Float32),
main_type_area_coverage Nullable(Float32),
estate_charge_coverage Nullable(Float32),
finish_building_date_coverage Nullable(Float32),
all_area_coverage Nullable(Float32),
build_area_coverage Nullable(Float32),
greening_rate_coverage Nullable(Float32),
volume_rate_coverage Nullable(Float32),
consumption_rating_coverage Nullable(Float32)
)
engine = Memory;
+22
View File
@@ -0,0 +1,22 @@
CREATE TABLE tmp.community_summary
(
_id String COMMENT '小区ID',
city String COMMENT '城市',
region String COMMENT '区域',
name String COMMENT '名称',
lng Nullable(Float32) COMMENT '经度',
lat Nullable(Float32) COMMENT '纬度',
alias Array(String) COMMENT '别名',
estate_type2 String COMMENT '物业类型',
build_type String COMMENT '建筑类型',
household_count String COMMENT '户数',
block String COMMENT '板块',
estate_charge String COMMENT '物业费',
finish_building_date String COMMENT '建成年代',
m_date DateTime('Asia/Shanghai') default now() COMMENT '更新时间',
avg_price Nullable(Int32) COMMENT '均价',
total_price Nullable(Int32) COMMENT '户均总价',
price_rating Nullable(Int32) COMMENT '价格评级',
consumption_rating Nullable(Int32) COMMENT '财力评级'
) ENGINE ReplacingMergeTree(m_date)
PARTITION BY city PRIMARY KEY (city, region, name) ORDER BY (city, region, name);
+43
View File
@@ -0,0 +1,43 @@
CREATE TABLE tmp.mt_community
(
id String COMMENT '小区ID',
city String COMMENT '城市',
region String COMMENT '区域',
name String COMMENT '名称',
alias String COMMENT '别名',
address String COMMENT '地址',
addr_alias String COMMENT '地址别名',
avg_price Nullable(INT) COMMENT '均价',
rent_price Nullable(INT) COMMENT '平米租金',
lng Nullable(Float32) COMMENT '经度',
lat Nullable(Float32) COMMENT '纬度',
block String COMMENT '板块',
loop_line_info String COMMENT '环线',
business_quarter String COMMENT '商圈',
households String COMMENT '户数',
property_fee String COMMENT '物业费',
car_site_count String COMMENT '停车位',
property_life String COMMENT '产权年限',
house_property String COMMENT '产权性质',
all_area String COMMENT '占地面积',
build_area String COMMENT '建筑面积',
plot_ratio String COMMENT '容积率',
landscaping_ratio String COMMENT '绿化率',
property_type String COMMENT '物业类型',
build_type String COMMENT '建筑类型',
build_structure String COMMENT '建筑结构',
finish_building_date String COMMENT '建成年代',
developer String COMMENT '开发商',
house_type String COMMENT '户型',
house_area String COMMENT '房型建筑面积',
deal_num Nullable(INT) COMMENT '近一年成交量',
district_facilitys String COMMENT '周边配套数量字段',
school Nullable(INT) COMMENT '学校数量',
market Nullable(INT) COMMENT '商场数量',
subway Nullable(INT) COMMENT '地铁数量',
bank Nullable(INT) COMMENT '银行数量',
hospital Nullable(INT) COMMENT '医院数量',
status INT COMMENT '0 正常 1 删除',
m_date DateTime('Asia/Shanghai') default now() COMMENT '更新时间'
) ENGINE ReplacingMergeTree(m_date)
PARTITION BY city PRIMARY KEY (city, region, name) ORDER BY (city, region, name);
+3
View File
@@ -0,0 +1,3 @@
pip uninstall goojia_utils
pip install goojia_utils -i https://repo.fangjia.com/repository/pypi-public/simple/
pip install -r /opt/airflow/dags/repo/dags/requirements.txt -i https://repo.fangjia.com/repository/pypi-public/simple/
-3
View File
@@ -1,3 +0,0 @@
"""
建行脚本
"""
+16
View File
@@ -0,0 +1,16 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="evaluate_dag",
schedule="30 16 27 * *",
start_date=pendulum.datetime(2023, 3, 28, tz="Asia/Shanghai"),
catchup=True,
description="每月27日估值文件检测"
) as dag:
t1 = BashOperator(
task_id='evaluate',
bash_command='python /opt/airflow/dags/repo/dags/jh/merge/evaluate.py {{dag_run.conf["day"] if "day" in dag_run.conf}}',
)
t1
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="merge_dag",
# 每日下午4点
schedule="0 16 * * *",
start_date=pendulum.datetime(2023, 3, 18, tz="Asia/Shanghai"),
catchup=True,
description="每日融合文件检测, 若想指定检测某一天如20230314,则传参day='20230314'"
) as dag:
t1 = BashOperator(
task_id='merge',
bash_command='python /opt/airflow/dags/repo/dags/jh/merge/merge.py {{dag_run.conf["day"] if "day" in dag_run.conf}}',
)
t1
@@ -1,12 +1,12 @@
from airflow.operators.bash import BashOperator from airflow.providers.standard.operators.bash import BashOperator
import pendulum import pendulum
from airflow import DAG from airflow import DAG
with DAG( with DAG(
dag_id="price_exception_dag", dag_id="price_exception_dag",
# 每月23日运行 # 每月23日运行
schedule_interval="0 10 23 * *", schedule="0 10 23 * *",
start_date=pendulum.datetime(2022, 8, 23, tz="Asia/Shanghai"), start_date=pendulum.datetime(2023, 3, 17, tz="Asia/Shanghai"),
catchup=True, catchup=True,
description="建行每月价格异常检测脚本,默认检测本月与上月差异, 若对比上月与前月价格,设置year_month=\"202210\"(前月日期即可) 若输出样本存在与否,设置year_month=sample" description="建行每月价格异常检测脚本,默认检测本月与上月差异, 若对比上月与前月价格,设置year_month=\"202210\"(前月日期即可) 若输出样本存在与否,设置year_month=sample"
) as dag: ) as dag:
+17
View File
@@ -0,0 +1,17 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="samples_notice_dag",
# 每月10日1点运行
schedule="0 0 10 * *",
start_date=pendulum.datetime(2025, 6, 9, tz="Asia/Shanghai"),
catchup=True,
description="样本数量监控提醒"
) as dag:
t1 = BashOperator(
task_id='samples_notice',
bash_command='python /opt/airflow/dags/repo/dags/jh/sample/samples_notice.py',
)
t1
+16
View File
@@ -0,0 +1,16 @@
from airflow.providers.standard.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="third_gxd_dag",
schedule=None,
start_date=pendulum.datetime(2024, 7, 22, tz="Asia/Shanghai"),
catchup=True,
description="第三方均价及国信达价格输出"
) as dag:
t1 = BashOperator(
task_id='gxd_dag',
bash_command='python /opt/airflow/dags/repo/dags/jh/price/price_third_gxd.py {{dag_run.conf["yearmonth"] if "yearmonth" in dag_run.conf}} ',
)
t1
@@ -1,12 +1,12 @@
from airflow.operators.bash import BashOperator from airflow.providers.standard.operators.bash import BashOperator
import pendulum import pendulum
from airflow import DAG from airflow import DAG
with DAG( with DAG(
dag_id="third_price_dag", dag_id="third_price_dag",
# 每月23日运行 # 每月23日运行
schedule_interval="0 16 18 * *", schedule="0 16 18 * *",
start_date=pendulum.datetime(2022, 11, 21, tz="Asia/Shanghai"), start_date=pendulum.datetime(2023, 3, 17, tz="Asia/Shanghai"),
catchup=True, catchup=True,
description="建行过滤第三方均价" description="建行过滤第三方均价"
) as dag: ) as dag:
View File
+131
View File
@@ -0,0 +1,131 @@
#!/usr/bin/env python
# coding: utf-8
from goojia_utils import dateutils, ftputils, tarutils
import datetime
import shutil
import os
import requests
import logging
# #### 检测传输压缩文件是否完整
# ##### 2.估值每月27日检测
# In[60]:
log = logging.getLogger(__name__)
date = dateutils.format_date(datetime.datetime.now(), '%Y%m') + '28'
path = '/home/airflow/data/gz'
local_aim_dir = path + '/' + date
if not os.path.exists(path):
os.makedirs(path)
if not os.path.exists(local_aim_dir):
os.makedirs(local_aim_dir)
client = ftputils.SftpClient()
client.sftp_connect(username="ccb", password="ccb!@#", host="192.168.0.45", port=17495)
client.get(remote="/home/ccb_evaluation/output/" + date + '/WS_GJ_000000_' + date + '_0001.tar', local=local_aim_dir)
client.get(remote="/home/ccb_evaluation/output/" + date + '/' + date + '.ok', local=path + '/' + date)
client.get(remote="/home/ccb_evaluation/output/" + date + '/WS_GJ_000000_' + date + '_0002.tar', local=local_aim_dir)
client.disconnect()
# In[57]:
## 解压
result = []
try:
dirs = os.listdir(local_aim_dir)
if date + '.ok' not in dirs:
result.append(date + '.ok' + '文件不存在')
tar_file = 'WS_GJ_000000_' + date + '_0001.tar'
has_two_tar = False
if 'WS_GJ_000000_' + date + '_0002.tar' in dirs:
has_two_tar = True
shutil.rmtree(local_aim_dir + '/' + date + '_2')
tarutils.untar_file(local_aim_dir + '/WS_GJ_000000_' + date + '_0002.tar', local_aim_dir + '/')
if tar_file not in dirs:
result.append(tar_file + '文件不存在')
else:
if has_two_tar is True:
shutil.rmtree(local_aim_dir + '/' + date + '_1')
tarutils.untar_file(local_aim_dir + "/" + tar_file, local_aim_dir + '/')
if has_two_tar is False:
sub_dirs = os.listdir(local_aim_dir)
else:
sub_dirs = os.listdir(local_aim_dir + '/' + date + '_1')
sub_dirs.extend(os.listdir(local_aim_dir + '/' + date + '_2'))
if 'MODEL_000000_000000_AVM_EVALUATION_GJNB_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_AVM_EVALUATION_GJNB_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_AVM_EVALUATION_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_AVM_EVALUATION_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_AVM_EVALUATION_ITEM_GJNB_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_AVM_EVALUATION_ITEM_GJNB_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_AVM_EVALUATION_ITEM_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_AVM_EVALUATION_ITEM_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_COMMBUILDTYPE_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_COMMBUILDTYPE_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_COMMUNITY_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_COMMUNITY_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_DEAL_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_DEAL_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_DISTRICT_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_DISTRICT_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_FORSALE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_FORSALE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_POI_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_POI_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_CITY_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_CITY_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_COMMUNITY_BASE_PRICE_GJNB_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_COMMUNITY_BASE_PRICE_GJNB_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_COMMUNITY_BASE_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_COMMUNITY_BASE_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_COMMUNITY_GUIDE_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_COMMUNITY_GUIDE_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_COMMUNITY_STAGING_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_COMMUNITY_STAGING_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_DISTRICT_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_DISTRICT_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_EVALUATE_RATE_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_EVALUATE_RATE_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_GUIDED_PRICE_GJ_ALL_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_GUIDED_PRICE_GJ_ALL_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_COMMUNITY_P_DESCRIBE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_COMMUNITY_P_DESCRIBE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'MODEL_000000_000000_COMM_CASE_DETAIL_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('MODEL_000000_000000_COMM_CASE_DETAIL_GJ_ADD_' + date + '_0001.dat文件不存在')
if len(result) == 0:
result.append('估值检测完毕未发现异常文件')
shutil.rmtree(local_aim_dir)
else:
result.append(" @高娟 ")
except Exception as ex:
result.append(" @高娟 ")
result.append(str(ex))
infos = "【" + date + "自动检测】"
for item in result:
infos += item + "\n"
data = {
"msgtype": "text",
"text": {
"content": infos
}
}
log.info(data)
# 发送消息至群
requests.post(url='https://oapi.dingtalk.com/robot/send?access_token=1b6d0d782d3c316f2d3b6b18ea5144816170dd392befbf2042a24a01fb84b736',
json=data)
+28 -37
View File
@@ -1,30 +1,23 @@
#!/usr/bin/env python
# coding: utf-8
# In[56]:
from goojia_utils import dateutils, ftputils, tarutils from goojia_utils import dateutils, ftputils, tarutils
import datetime import datetime
import calendar
import requests import requests
import json
import shutil import shutil
import os import os
import sys
import logging
log = logging.getLogger(__name__)
result = [] result = []
# #### 检测传输压缩文件是否完整
# ##### 1. 融合每天检测 # ##### 1. 融合每天检测
try: try:
today = dateutils.add_months(datetime.datetime.now(), days=1) date = sys.argv[1] if len(sys.argv) > 1 else None
date = dateutils.format_date(today, '%Y%m%d') if date is None:
today = dateutils.add_months(datetime.datetime.now(), days=1)
date = dateutils.format_date(today, '%Y%m%d')
print('当前月:' + date) print('当前月:' + date)
curr_month_days = calendar.monthrange(today.year, today.month)[1] # curr_month_days = calendar.monthrange(today.year, today.month)[1]
print('当月天数:' + str(curr_month_days)) # print('当月天数:' + str(curr_month_days))
# In[79]: # In[79]:
@@ -121,41 +114,38 @@ try:
result.append('FJW_000000_000000_BIGDATA_DISTPRICE_FJW_ADD_' + date + '_0001.dat文件不存在') result.append('FJW_000000_000000_BIGDATA_DISTPRICE_FJW_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_COMMPRICE_FJW_ADD_' + date + '_0001.dat' not in sub_dirs: if 'FJW_000000_000000_BIGDATA_COMMPRICE_FJW_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_BIGDATA_COMMPRICE_FJW_ADD_' + date + '_0001.dat文件不存在') result.append('FJW_000000_000000_BIGDATA_COMMPRICE_FJW_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs: if 'FJW_000000_000000_EVALUATE_RATE_CITY_FJW_HIS_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在') result.append('FJW_000000_000000_EVALUATE_RATE_CITY_FJW_HIS_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_EVALUATE_RATE_COMM_FJW_HIS_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_EVALUATE_RATE_COMM_FJW_HIS_ADD_' + date + '_0001.dat文件不存在')
# if 'FJW_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
# result.append('FJW_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_MODELING_BASE_ADD_' + date + '_0001.dat' not in sub_dirs: if date.endswith('28'):
result.append('FJW_000000_000000_BIGDATA_MODELING_BASE_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_SUPPLIER_STATISTICS_ADD_' + date + '_0001.dat' not in sub_dirs: if 'FJW_000000_000000_BIGDATA_SUPPLIER_STATISTICS_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_BIGDATA_SUPPLIER_STATISTICS_ADD_' + date + '_0001.dat文件不存在') result.append('FJW_000000_000000_BIGDATA_SUPPLIER_STATISTICS_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_BIGDATA_THIRD_PARTY_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_COMM_BASE_PRICE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_COMM_BASE_PRICE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_COMM_DESCRIBE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_COMM_DESCRIBE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_MODELING_BASE_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_BIGDATA_MODELING_BASE_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_GUIDE_STATISTICS_ADD_' + date + '_0001.dat' not in sub_dirs: if 'FJW_000000_000000_GUIDE_STATISTICS_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_GUIDE_STATISTICS_ADD_' + date + '_0001.dat文件不存在') result.append('FJW_000000_000000_GUIDE_STATISTICS_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_COMMUNITY_P_DESCRIBE_GJ_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_COMMUNITY_P_DESCRIBE_GJ_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_COMMUNITY_BASE_PRICE_GJNB_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_COMMUNITY_BASE_PRICE_GJNB_ADD_' + date + '_0001.dat文件不存在')
if 'FJW_000000_000000_BIGDATA_FORSALE_FJW_ADD_' + date + '_0001.dat' not in sub_dirs:
result.append('FJW_000000_000000_BIGDATA_FORSALE_FJW_ADD_' + date + '_0001.dat文件不存在')
if len(result) == 0: if len(result) == 0:
result.append('融合检测完毕未发现异常文件') result.append('融合检测完毕未发现异常文件')
shutil.rmtree(path + "/" + date) shutil.rmtree(path + "/" + date)
else:
result.append(" @高娟 ")
except Exception as ex: except Exception as ex:
result.append(" @高娟 ")
result.append(str(ex)) result.append(str(ex))
# In[82]:
print(result)
# In[76]:
infos = "【" + date + "自动检测】" infos = "【" + date + "自动检测】"
for item in result: for item in result:
infos += item + "\n" infos += item + "\n"
@@ -165,6 +155,7 @@ data = {
"content": infos "content": infos
} }
} }
log.info(data)
# 发送消息至群 # 发送消息至群
requests.post(url='https://oapi.dingtalk.com/robot/send?access_token=1b6d0d782d3c316f2d3b6b18ea5144816170dd392befbf2042a24a01fb84b736', requests.post(url='https://oapi.dingtalk.com/robot/send?access_token=1b6d0d782d3c316f2d3b6b18ea5144816170dd392befbf2042a24a01fb84b736',
json=data) json=data)
-17
View File
@@ -1,17 +0,0 @@
from airflow.operators.bash import BashOperator
import pendulum
from airflow import DAG
with DAG(
dag_id="merge_dag",
# 每日下午4点
schedule_interval="0 16 * * *",
start_date=pendulum.datetime(2022, 11, 15, tz="Asia/Shanghai"),
catchup=True,
description="每日融合文件检测"
) as dag:
t1 = BashOperator(
task_id='merge',
bash_command='python /opt/airflow/dags/repo/dags/jh/merge/merge.py',
)
t1
View File
+109
View File
@@ -0,0 +1,109 @@
import pandas as pd
city_name_id_map = {'七台河': '230900', '万宁': '469006', '三亚': '460200', '东莞': '441900', '三明': '350400', '三沙': '460300', '三门峡': '411200',
'上海': '310000', '上饶': '361100', '东方': '469007', '中山': '442000', '东营': '370500', '中卫': '640500', '临夏': '622900',
'临汾': '141000', '临沂': '371300', '临沧': '530900', '临高': '469024', '丹东': '210600', '丽水': '331100', '丽江': '530700',
'乌兰察布': '150900', '乌海': '150300', '乌鲁木齐': '650100', '乐东': '469027', '乐山': '511100', '九江': '360400',
'云浮': '445300', '五家渠': '659004', '五指山': '469001', '亳州': '341600', '仙桃': '429004', '伊春': '230700',
'伊犁': '654000', '佛山': '440600', '佳木斯': '230800', '保亭': '469029', '保定': '130600', '保山': '530500', '信阳': '411500',
'儋州': '460400', '克州': '653000', '克拉玛依': '650200', '六安': '341500', '六盘水': '520200', '兰州': '620100',
'兴安': '152200', '内江': '511000', '凉山': '513400', '包头': '150200', '北京': '110000', '北屯': '659005', '北海': '450500',
'十堰': '420300', '南京': '320100', '南充': '511300', '南宁': '450100', '南平': '350700', '南昌': '360100', '南通': '320600',
'南阳': '411300', '博州': '652700', '厦门': '350200', '双河': '659007', '双鸭山': '230500', '可克达拉': '659008',
'台州': '331000', '合肥': '340100', '吉安': '360800', '吉林': '220200', '吐鲁番': '650400', '吕梁': '141100',
'吴忠': '640300', '周口': '411600', '呼伦贝尔': '150700', '呼和浩特': '150100', '和田': '653200', '咸宁': '421200',
'咸阳': '610400', '哈密': '650500', '哈尔滨': '230100', '唐山': '130200', '商丘': '411400', '商洛': '611000',
'喀什': '653100', '嘉兴': '330400', '嘉峪关': '620200', '四平': '220300', '固原': '640400', '图木舒克': '659003',
'塔城': '654200', '大兴安岭': '232700', '大同': '140200', '大庆': '230600', '大理': '532900', '大连': '210200',
'天水': '620500', '天津': '120000', '天门': '429006', '太原': '140100', '威海': '371000', '娄底': '431300', '孝感': '420900',
'宁德': '350900', '宁波': '330200', '安庆': '340800', '安康': '610900', '安阳': '410500', '安顺': '520400', '定安': '469021',
'定西': '621100', '宜宾': '511500', '宜昌': '420500', '宜春': '360900', '宝鸡': '610300', '宣城': '341800', '宿州': '341300',
'宿迁': '321300', '屯昌': '469022', '山南': '540500', '岳阳': '430600', '崇左': '451400', '巴中': '511900',
'巴彦淖尔': '150800', '巴州': '652800', '常州': '320400', '常德': '430700', '平凉': '620800', '平顶山': '410400',
'广元': '510800', '广安': '511600', '广州': '440100', '庆阳': '621000', '廊坊': '131000', '延安': '610600',
'延边': '222400', '开封': '410200', '张家口': '130700', '张家界': '430800', '张掖': '620700', '徐州': '320300',
'德宏': '533100', '德州': '371400', '德阳': '510600', '忻州': '140900', '怀化': '431200', '怒江': '533300', '恩施': '422800',
'惠州': '441300', '成都': '510100', '扬州': '321000', '承德': '130800', '抚州': '361000', '抚顺': '210400', '拉萨': '540100',
'揭阳': '445200', '攀枝花': '510400', '文山': '532600', '文昌': '469005', '新乡': '410700', '新余': '360500',
'无锡': '320200', '日喀则': '540200', '日照': '371100', '昆明': '530100', '昆玉': '659009', '昌吉': '652300',
'昌江': '469026', '昌都': '540300', '昭通': '530600', '晋中': '140700', '晋城': '140500', '普洱': '530800', '景德镇': '360200',
'曲靖': '530300', '朔州': '140600', '朝阳': '211300', '本溪': '210500', '来宾': '451300', '杭州': '330100', '松原': '220700',
'林芝': '540400', '果洛': '632600', '枣庄': '370400', '柳州': '450200', '株洲': '430200', '桂林': '450300', '梅州': '441400',
'梧州': '450400', '楚雄': '532300', '榆林': '610800', '武威': '620600', '武汉': '420100', '毕节': '520500', '永州': '431100',
'汉中': '610700', '汕头': '440500', '汕尾': '441500', '江门': '440700', '池州': '341700', '沈阳': '210100', '沧州': '130900',
'河池': '451200', '河源': '441600', '泉州': '350500', '泰安': '370900', '泰州': '321200', '泸州': '510500', '洛阳': '410300',
'济南': '370100', '济宁': '370800', '济源': '419001', '海东': '630200', '海北': '632200', '海南州': '632500',
'海口': '460100', '海西': '632800', '淄博': '370300', '淮北': '340600', '淮南': '340400', '淮安': '320800', '深圳': '440300',
'清远': '441800', '温州': '330300', '渭南': '610500', '湖州': '330500', '湘潭': '430300', '湘西': '433100', '湛江': '440800',
'滁州': '341100', '滨州': '371600', '漯河': '411100', '漳州': '350600', '潍坊': '370700', '潜江': '429005', '潮州': '445100',
'澄迈': '469023', '濮阳': '410900', '烟台': '370600', '焦作': '410800', '牡丹江': '231000', '玉林': '450900',
'玉树': '632700', '玉溪': '530400', '珠海': '440400', '琼中': '469030', '琼海': '469002', '甘南': '623000', '甘孜': '513300',
'白城': '220800', '白山': '220600', '白沙': '469025', '白银': '620400', '百色': '451000', '益阳': '430900', '盐城': '320900',
'盘锦': '211100', '眉山': '511400', '石嘴山': '640200', '石家庄': '130100', '石河子': '659001', '神农架': '429021',
'福州': '350100', '秦皇岛': '130300', '红河': '532500', '绍兴': '330600', '绥化': '231200', '绵阳': '510700',
'聊城': '371500', '肇庆': '441200', '自贡': '510300', '舟山': '330900', '芜湖': '340200', '苏州': '320500', '茂名': '440900',
'荆州': '421000', '荆门': '420800', '菏泽': '371700', '莆田': '350300', '萍乡': '360300', '营口': '210800',
'葫芦岛': '211400', '蚌埠': '340300', '衡水': '131100', '衡阳': '430400', '衢州': '330800', '襄阳': '420600', '西双版纳': '532800',
'西宁': '630100', '西安': '610100', '许昌': '411000', '贵港': '450800', '贵阳': '520100', '贺州': '451100', '资阳': '512000',
'赣州': '360700', '赤峰': '150400', '辽源': '220400', '辽阳': '211000', '达州': '511700', '运城': '140800', '连云港': '320700',
'迪庆': '533400', '通化': '220500', '通辽': '150500', '遂宁': '510900', '遵义': '520300', '邢台': '130500', '那曲': '540600',
'邯郸': '130400', '邵阳': '430500', '郑州': '410100', '郴州': '431000', '鄂尔多斯': '150600', '鄂州': '420700', '酒泉': '620900',
'重庆': '500000', '金华': '330700', '金昌': '620300', '钦州': '450700', '铁岭': '211200', '铁门关': '659006', '铜仁': '520600',
'铜川': '610200', '铜陵': '340700', '银川': '640100', '锡林郭勒': '152500', '锦州': '210700', '镇江': '321100', '长春': '220100',
'长沙': '430100', '长治': '140400', '阜新': '210900', '阜阳': '341200', '防城港': '450600', '阳江': '441700', '阳泉': '140300',
'阿克苏': '652900', '阿勒泰': '654300', '阿坝': '513200', '阿拉善': '152900', '阿拉尔': '659002', '阿里': '542500', '陇南': '621200',
'陵水': '469028', '随州': '421300', '雅安': '511800', '青岛': '370200', '鞍山': '210300', '韶关': '440200',
'驻马店': '411700', '鸡西': '230300', '鹤壁': '410600', '鹤岗': '230400', '鹰潭': '360600', '黄冈': '421100', '黄南': '632300',
'黄山': '341000', '黄石': '420200', '黑河': '231100', '黔东南': '522600', '黔南': '522700', '黔西南': '522300', '齐齐哈尔': '230200',
'龙岩': '350800', "马鞍山": "340500"}
third_data_source_id_map = {'01': '诸葛找房','02': '安居客','03': '房天下','04': '链家','05': '城市居家',
'06': '麦田','07': '贝壳','08': '淘房356','09': '中原','10': '兔博士',
'11': 'Q房','12': '房探','13': '房多多','14': '我爱我家','15': '禧泰数据',
'16': '吉屋网','17': '乐居','18': '21世纪','19': '房产超市','20': '58同城'}
city_id_name_map = {k: v for v, k in city_name_id_map.items()}
def get_city_name(city_id):
"""
通过城市ID获取城市名
:param city_id:
:return:
"""
return city_id_name_map[city_id] if city_id in city_id_name_map else None
def get_city_id(city_name):
"""
通过城市名获取城市ID
:param city_name:
:return:
"""
return city_name_id_map[city_name] if city_name in city_name_id_map else None
def is_city_id(city):
"""
判断是不是合法的cityid
:param city:
:return:
"""
return city in city_id_name_map
def is_city_name(city):
"""
判断是不是合法的城市名
:param city:
:return:
"""
return city in city_name_id_map
def get_city_data():
"""
返回所有的城市数据DataFrame[cityid,cityname]
:return:
"""
data = list()
for city_id in city_id_name_map.keys():
data.append({"cityid": city_id, "cityname": city_id_name_map[city_id]})
return pd.DataFrame(data)
+25
View File
@@ -0,0 +1,25 @@
from pymongo import MongoClient
def getMongo(*args):
if len(args) == 6:
con = MongoClient(args[0], args[1]) # 获取链接\
db_auth = con.fangjia_craw
if args[2] =="vip_site":
db_auth.authenticate(args[4], args[5],"fangjia_craw")
else:
db_auth.authenticate(args[4], args[5],"admin")
db = con[args[2]] # 关联数据库
collection = db[args[3]] # 关联数据表
elif len(args) == 4:
con = MongoClient(args[0], args[1]) # 获取链接
db = con[args[2]] # 关联数据库
collection = db[args[3]] # 关联数据表
return collection
#线上小区库
seaweed_on = getMongo("192.168.0.43",28017,"fangjia","seaweed","developer","G00IDCDEV12385")
sz_price_on = getMongo("192.168.0.43",28017,"fangjia","sz_price","developer","G00IDCDEV12385")
district_stat_on = getMongo("192.168.0.43",28017,"fangjia_stat","district_stat","developer","G00IDCDEV12385")
seaweed_map = getMongo("192.168.0.235",27018,"fangjia","seaweedmap")
web_community_check = getMongo("192.168.0.235",27018,"fangjia","web_community_check")
web_invalid_comm = getMongo("192.168.0.235",27018,"fangjia","web_invalid_comm")
addr_nine_level = getMongo("192.168.0.206",27020,"etl","nine_level")
gxd_price = getMongo("192.168.0.43",28017,"fangjia","confirm_price_data","developer","G00IDCDEV12385")
+81
View File
@@ -0,0 +1,81 @@
from mongo_client import seaweed_on,seaweed_map,addr_nine_level,district_stat_on,gxd_price
import pandas as pd
import datetime
import pytz
pd.set_option ('display.max_colwidth',5)
def parse_date(time, fm='%Y-%m-%d'):
pacific = pytz.timezone('Asia/Shanghai')
return pacific.localize(datetime.datetime.strptime(time, fm))
def get_last_weekend():
sunday = datetime.datetime.utcnow()
sunday = sunday.replace(hour=0, minute=0, second=0, microsecond=0)
one_day = datetime.timedelta(days=1)
while sunday.weekday() != 6:
sunday -= one_day
pacific = pytz.timezone('Asia/Shanghai')
return pacific.localize(sunday)
confirm_date = parse_date("2024-05-20")
def get_seaweed(city=None,status=0):
seaweed_fields = {"city": 1, "region": 1, "name": 1,'estate_type2':1,'cat':1,"address":1,"alias":1,"lat":1,"lng":1,"addr_alias":1,"quality":1,"name_cat":1,"mapping_alias":1,"m_date":1,"is_new":1}
# seaweed_fields = {"_id": 0}
if city:
if city in ['阿坝','玉树','甘孜','德宏','迪庆','海北','海西','黄南','海南','怒江','文山','西双版纳','凉山','果洛']:
city = city+'州'
elif city in ['锡林郭勒','兴安','阿拉善']:
city = city+'盟'
elif city=='克州':
city = '克孜勒苏'
elif city=='博州':
city = '博尔塔拉'
elif city=='巴州':
city = '巴音郭楞'
seaweed_condition = {"cat": "district", "status": status, 'city':city}
else:
seaweed_condition = {"cat": "district", "status": status}
seaweed_data = seaweed_on.find(seaweed_condition, seaweed_fields)
seaweed_df = pd.DataFrame(seaweed_data)
seaweed_df = seaweed_df.rename(columns={'estate_type2': 'property_type'})
return seaweed_df
def get_map(city=None):
# seaweed_fields = {"city": 1, "region": 1, "name": 1,"address":1,"alias":1,"lat":1,"lng":1,"estate_type2":1,"addr_alias":1}
seaweed_fields = {"_id": 0}
if city:
seaweed_condition = {'city':city}
else:
seaweed_condition = {}
seaweed_map_data = seaweed_map.find(seaweed_condition, seaweed_fields)
seaweed_map_df = pd.DataFrame(seaweed_map_data)
return seaweed_map_df
def get_addr_nine_level(count):
_fields = {"address":1,"province":1,"city": 1, "district": 1,"township": 1, "name": 1,"":1}
_condition = {'address':{"$exists": True, "$ne": None},"remark":{"$exists": False}}
addr_nine_data = addr_nine_level.find(_condition,_fields).limit(count)
addr_nine_df = pd.DataFrame(addr_nine_data)
return addr_nine_df
def get_district_stat_on(city=None):
price_date = get_last_weekend()
stat_fields = {"_id": 0, "city": 1, "region": 1, "district_name": 1, "avg_price": 1,"avg_price_2b":1}
if city:
stat_condition = {'city':city,"weekend": price_date}
else:
stat_condition = {"weekend": price_date}
stat_data = district_stat_on.find(stat_condition, stat_fields)
stat_df = pd.DataFrame(stat_data)
stat_df = stat_df.rename(columns={'district_name': 'name'})
return stat_df
def get_gxd_price():
gxd_fields = {"_id": 0}
gxd_condition = {'source':'GXD'}
gxd_price_data = gxd_price.find(gxd_condition, gxd_fields)
gxd_price_df = pd.DataFrame(gxd_price_data)
return gxd_price_df
if __name__ == '__main__':
city = '厦门'
df = get_gxd_price()
print(df.head())
+4 -9
View File
@@ -3,12 +3,7 @@ import logging
import pandas as pd import pandas as pd
import requests import requests
from datetime import datetime from datetime import datetime
from goojia_utils import dateutils, oracle from goojia_utils import dateutils, pgsql
from sqlalchemy import create_engine
os.environ["NLS_LANG"] = "American_America.AL32UTF8"
os.environ["ORACLE_HOME"] = "/home/airflow/instantclient_21_7"
engine = create_engine("postgresql+psycopg2://postgres:postgres@192.168.0.236:5432/warehouse", echo=False)
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
path = "/home/airflow/data/jh/price" path = "/home/airflow/data/jh/price"
@@ -17,9 +12,9 @@ if not os.path.exists(path):
year_month = int(sys.argv[1]) if len(sys.argv) > 1 else int(dateutils.format_date(datetime.utcnow(), "%Y%m")) year_month = int(sys.argv[1]) if len(sys.argv) > 1 else int(dateutils.format_date(datetime.utcnow(), "%Y%m"))
log.info(f"测试月份:{year_month}") log.info(f"测试月份:{year_month}")
avm_df = pd.read_sql(f"select urbn_id,urbn_nm,gujia_bds_comm_num,gujiabdsvalmnycommnum from t_app_p9pha_plda.avm_evaluation_gj_h where gujia_bds_yrmo={year_month} ", con=engine) avm_df = pgsql.scan_by_sql_to_data_frame(f"select urbn_id,urbn_nm,gujia_bds_comm_num,gujiabdsvalmnycommnum from price_stat.avm_evaluation_gj_h where gujia_bds_yrmo={year_month} ")
comm_df = pd.read_sql(f"select urbn_id,gujia_admndiv_nm,gujia_comm_id,comm_nm from base.t0831_bigdata_community_gj_h where gujia_bds_bds_st=0 ", con=engine) comm_df = pgsql.scan_by_sql_to_data_frame(f"select urbn_id,gujia_admndiv_nm,gujia_comm_id,comm_nm from base.bigdata_community where gujia_bds_bds_st=0 ")
price_df = pd.read_sql(f"select gujia_bds_id as gujia_comm_id,gujia_comm_basic_prc from t_app_p9pha_plda.avm_community_base_price_gj_h where gujiabds_clc_yrmo_val={year_month} ", con=engine) price_df = pgsql.scan_by_sql_to_data_frame(f"select gujia_bds_id as gujia_comm_id,gujia_comm_basic_prc from price_stat.avm_community_base_price where gujiabds_clc_yrmo_val={year_month} ")
comm_df = comm_df.merge(avm_df, on="urbn_id", how="left") comm_df = comm_df.merge(avm_df, on="urbn_id", how="left")
comm_df = comm_df.merge(price_df, on="gujia_comm_id", how="left") comm_df = comm_df.merge(price_df, on="gujia_comm_id", how="left")
no_price_comm_df = comm_df[comm_df.gujia_comm_basic_prc.isna()][['urbn_id','urbn_nm','gujia_admndiv_nm','gujia_comm_id','comm_nm']] no_price_comm_df = comm_df[comm_df.gujia_comm_basic_prc.isna()][['urbn_id','urbn_nm','gujia_admndiv_nm','gujia_comm_id','comm_nm']]
+42
View File
@@ -0,0 +1,42 @@
import os
import pandas as pd
from pandas import DataFrame
from retrying import retry
from sqlalchemy import create_engine
# os.environ["NLS_LANG"] = "AMERICAN_AMERICA.AL32UTF8"
class PgsqlDb:
"""
PgsqlDb Database Base Class
"""
def __init__(self,db):
if db == 'warehouse':
url=f"postgresql+psycopg2://postgres:postgres@192.168.0.236:5433/{db}"
elif db == 'fangjia':
url = f"postgresql+psycopg2://postgres:Goojia123456@192.168.0.235:5433/{db}"
self.engine = create_engine(url, echo=False,client_encoding='utf-8')
@retry(stop_max_attempt_number=3, wait_random_min=1000, wait_random_max=3000)
def read(self, sql) -> DataFrame:
"""
读取数据
:param sql:
:return:
"""
data_df = pd.read_sql(sql, con=self.engine)
return data_df
@retry(stop_max_attempt_number=3, wait_random_min=1000, wait_random_max=3000)
def count(self, sql) -> int:
"""
执行统计类SQL
:param sql: eg:select count(1) from t
:return:
"""
return pd.read_sql_query(sql, con=self.engine).iat[0, 0]
def sql_execute(self,sql):
self.engine.execute(sql)
if __name__ == '__main__':
pgdb = PgsqlDb()
+100
View File
@@ -0,0 +1,100 @@
import pandas as pd
import os
from pgsql_client import PgsqlDb
os.environ['NLS_LANG'] = 'SAMPLIFIED CHINESE_CHINA.UTF8'
pgdb = PgsqlDb('warehouse')
pgdb_new = PgsqlDb('fangjia')
def get_community_base_price(cityid,yearmonth):
"""
查询提供建行小区数据
:param cr:
:return:
"""
sql = f"select gujia_admndiv_nm,gujia_bds_id,urbn_nm,comm_nm,gujiabds_clc_yrmo_val,gujia_comm_basic_prc,gujia_bds_clmtd from price_stat.avm_community_base_price where urbn_id='{cityid}' and gujiabds_clc_yrmo_val={yearmonth}"
data_df = pgdb.read(sql)
data_df = data_df.rename(columns={'gujia_admndiv_nm':'district','gujia_bds_id': '_id','urbn_nm':'city','comm_nm':'name','gujiabds_clc_yrmo_val':'yearmonth','gujia_comm_basic_prc':'price','gujia_bds_clmtd':'type'})
return data_df
def get_community_base_price_new(cityid,yearmonth):
"""
查询提供建行小区数据
:param cr:
:return:
"""
sql = f"select gujia_bds_id,urbn_nm,comm_nm,gujiabds_clc_yrmo_val,gujia_comm_basic_prc,gujia_bds_clmtd from price_stat.avm_community_base_price where urbn_id='{cityid}' and gujiabds_clc_yrmo_val={yearmonth}"
data_df = pgdb_new.read(sql)
data_df = data_df.rename(columns={'gujia_bds_id': '_id','urbn_nm':'city_copy','comm_nm':'name_copy','gujiabds_clc_yrmo_val':'yearmonth','gujia_comm_basic_prc':'price_new','gujia_bds_clmtd':'type_new'})
return data_df
def get_district_polyline(cityid=None,cr=None):
"""
查询提供建行小区数据
:param cr:
:return:
"""
if cityid:
sql = f"select * from bigdata_district_polyline_gd where cityid='{cityid}'"
else:
sql = f"select distinct city,distname from bigdata_district_polyline_gd"
queryse = cr.execute(sql)
columns = [i[0].lower() for i in queryse.description]
rs = cr.fetchall()
data_df = pd.DataFrame(rs,columns=columns)
return data_df
def get_community_ccb(cityid):
sql = f"select gujia_comm_id,gujia_admndiv_nm,comm_nm from base.bigdata_community where urbn_id='{cityid}' and gujia_bds_bds_st=0"
data_df = pgdb.read(sql)
data_df = data_df.rename(columns={'gujia_comm_id': '_id','gujia_admndiv_nm':'region','comm_nm':'name'})
return data_df
def get_bak_community(cityid,cr):
"""
查询小区数据
:param cr:
:return:
"""
sql = f"select a.*,b.longitude,b.latitude,c.distname from bigdata_community_bak a left join (select cityid, objectid, LONGITUDE, LATITUDE from bigdata_position_bak where objecttype = '5' and positiontype = '1') b on a.cityid = b.cityid and a.commid = b.objectid left join bigdata_district_bak c on a.cityid=c.cityid and a.distid=c.distid where a.cityid = '{cityid}'"
queryse = cr.execute(sql)
columns = [i[0].lower() for i in queryse.description]
rs = cr.fetchall()
data_df = pd.DataFrame(rs,columns=columns)
return data_df
def avm_price_copy(cityid,yearmonth):
sql = f"insert into price_stat.avm_community_base_price_gj_h_copy select * from price_stat.avm_community_base_price where urbn_id='{cityid}' and gujiabds_clc_yrmo_val={yearmonth}"
pgdb.sql_execute(sql)
def get_sample(cityid):
sql = f"select urbn_id,gujia_comm_id,comm_nm,gujia_bds_unitprc,guiabconstructionarea from base.bigdata_forsale_gj_h where urbn_id='{cityid}' and gujia_hs_src_dtsrc in ('1','4') and anc_tm >= TO_DATE('2023-04-01','YYYY-MM-DD')"
data_df = pgdb.read(sql)
return data_df
def get_deal(cityid):
sql = f"select urbn_id,gujia_comm_id,comm_nm,guiabconstructionarea,gujia_bds_trdtckt_prc from base.bigdata_deal where urbn_id='{cityid}' and txn_dt >= TO_DATE('2024-03-01','YYYY-MM-DD') and txn_dt < TO_DATE('2024-04-01','YYYY-MM-DD')"
data_df = pgdb.read(sql)
data_df = data_df.rename(columns={'urbn_id': 'cityid','gujia_comm_id': '_id','comm_nm': 'name', 'guiabconstructionarea': 'area','gujia_bds_trdtckt_prc': 'deal_price'})
return data_df
def get_deal_data(cityid):
sql = f"select * from base.bigdata_deal where urbn_id='{cityid}' and txn_dt >= TO_DATE('2024-03-01','YYYY-MM-DD') and txn_dt < TO_DATE('2024-04-01','YYYY-MM-DD')"
data_df = pgdb.read(sql)
return data_df
def get_region_price(cityid,yearmonth):
sql = f"select urbn_id,gujia_admndiv_id,gxda_admndiv_nm,gujia_bds_urbn_avprc from price_stat.avm_district_price_gj_h where urbn_id='{cityid}' and gujiabds_clc_yrmo_val={yearmonth}"
data_df = pgdb.read(sql)
data_df = data_df.rename(columns={'urbn_id': 'cityid','gujia_admndiv_id': 'district_id','gxda_admndiv_nm': 'district_name', 'gujia_bds_urbn_avprc': 'district_price'})
return data_df
def get_evaluation_third_price(cityid,yearmonth,type='第三方'):
sql = f"select urbn_id,gujia_comm_id,gujiabds_to_mkprc_mny from price_stat.avm_evaluation_item_gj_h where urbn_id='{cityid}' and gujiabds_clc_yrmo_val={yearmonth} and gujia_data_tp='{type}' "
data_df = pgdb.read(sql)
data_df = data_df.rename(columns={'urbn_id': 'cityid','gujiabds_to_mkprc_mny': 'third_price','gujia_comm_id':'_id'})
return data_df
def get_third_price(cityid,yearmonth):
sql = f"select urbn_id,gujia_bds_id,comm_nm,gujia_dtsr_dsc,gujia_bds_comm_avprc from base.bigdata_third_party_price where urbn_id='{cityid}' and gujiabds_avprc_dt_val={yearmonth} and gujia_dtsr_dsc in ('链家','安居客') "
data_df = pgdb.read(sql)
data_df = data_df.rename(columns={'urbn_id': 'cityid','gujia_bds_id': '_id','gujia_dtsr_dsc': 'source','gujia_bds_comm_avprc': 'price'})
return data_df
if __name__ == '__main__':
city='常州'
yearmonth=202302
get_community_base_price(city=city,yearmonth=yearmonth)
+27 -12
View File
@@ -2,13 +2,9 @@ import os, sys
import logging import logging
import pandas as pd import pandas as pd
import requests import requests
from datetime import datetime from goojia_utils import dateutils, pgsql
from goojia_utils import dateutils, oracle from datetime import datetime, timedelta
from sqlalchemy import create_engine from goojia_utils import MAIN_MONGO_CLIENT
os.environ["NLS_LANG"] = "American_America.AL32UTF8"
os.environ["ORACLE_HOME"] = "/home/airflow/instantclient_21_7"
engine = create_engine("postgresql+psycopg2://postgres:postgres@192.168.0.236:5432/warehouse", echo=False)
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
path = "/home/airflow/data/jh/price" path = "/home/airflow/data/jh/price"
@@ -27,13 +23,32 @@ if type == month_last:
month_last = month_before month_last = month_before
log.info(f"测试月份:{month_now}") log.info(f"测试月份:{month_now}")
price_df = pd.read_sql(f"select urbn_id,urbn_nm,gujia_admndiv_nm,gujia_bds_id as gujia_comm_id,comm_nm,gujia_bds_clmtd,gujia_comm_basic_prc from t_app_p9pha_plda.avm_community_base_price_gj_h where gujiabds_clc_yrmo_val={month_now} ", con=engine) price_df = pgsql.scan_by_sql_to_data_frame(f"select urbn_id,urbn_nm,gujia_admndiv_nm,gujia_bds_id as gujia_comm_id,comm_nm,gujia_bds_clmtd,gujia_comm_basic_prc from price_stat.avm_community_base_price where gujiabds_clc_yrmo_val={month_now} ")
deal_df = pd.read_sql(f"select gujia_bds_mdl_id,gujia_comm_id,gujia_bds_trdtckt_prc from base.t0831_bigdata_deal_gj_h where to_char(txn_dt,'yyyyMM')='{month_last}' ", con=engine) deal_df = pgsql.scan_by_sql_to_data_frame(f"select gujia_bds_mdl_id,gujia_comm_id,gujia_bds_trdtckt_prc from base.bigdata_deal where to_char(txn_dt,'yyyyMM')='{month_last}' ")
deal_price_df = deal_df.merge(price_df, on="gujia_comm_id", how="left") deal_price_df = deal_df.merge(price_df, on="gujia_comm_id", how="left")
deal_price_df["ppe"] = round((deal_price_df["gujia_comm_basic_prc"] - deal_price_df['gujia_bds_trdtckt_prc']) / deal_price_df['gujia_bds_trdtckt_prc'], 4) deal_price_df["ppe"] = round((deal_price_df["gujia_comm_basic_prc"] - deal_price_df['gujia_bds_trdtckt_prc']) / deal_price_df['gujia_bds_trdtckt_prc'], 4)
deal_price_df = deal_price_df[abs(deal_price_df['ppe']) > 0.15] #deal_price_df = deal_price_df[abs(deal_price_df['ppe']) > 0.15]
third_df = pgsql.scan_by_sql_to_data_frame(f"select gujia_bds_id as gujia_comm_id,gujia_dtsr_dsc,gujia_bds_comm_avprc from base.bigdata_third_party_price where gujiabds_avprc_dt_val={month_last} ")
third_df = third_df[third_df.gujia_dtsr_dsc.isin(['贝壳','链家'])]
# third_df = third_df[['gujia_comm_id','gujia_dtsr_dsc','gujia_bds_comm_avprc']]
pivot_df = third_df.pivot(index='gujia_comm_id', columns='gujia_dtsr_dsc', values='gujia_bds_comm_avprc').reset_index()
deal_price_df = pd.merge(deal_price_df, pivot_df, how='left', left_on='gujia_comm_id', right_on='gujia_comm_id')
# 获取上周日
today = datetime.today()
last_sunday = today - timedelta(days=today.weekday() + 1)
last_sunday = last_sunday.strftime("%Y-%m-%d")
stat = pd.DataFrame(list(MAIN_MONGO_CLIENT.get_collection("fangjia_stat", "district_stat").find({"weekend": dateutils.parse_date(last_sunday)},{"city":1,"region":1,"district_name":1,"avg_price":1,"avg_price_2b":1,"_id":0})))
seaweed = pd.DataFrame(list(MAIN_MONGO_CLIENT.get_collection("fangjia", "seaweed").find({"status":0, "cat":"district"},{'city':1, 'region':1,'name':1,'_id':1})))
comm_price = pd.merge(seaweed, stat, how='left', left_on=['city','region','name'], right_on=['city','region','district_name'])
comm_price = comm_price[['_id','avg_price','avg_price_2b']]
deal_price_df['gujia_comm_id'] = deal_price_df['gujia_comm_id'].astype('str')
comm_price['_id'] = comm_price['_id'].astype('str')
deal_price_df = pd.merge(deal_price_df, comm_price, how='left', left_on='gujia_comm_id', right_on='_id')
# third_df = pd.read_sql(f"select gujia_trdpt_avprc_id,gujia_bds_id as gujia_comm_id,gujia_dtsr_dsc,gujia_bds_comm_avprc from base.t0831_bigdata_third_party_price_gj_h where gujiabds_avprc_dt_val={month_last} ", con=engine)
# third_price_df = third_df.merge(price_df, on="gujia_comm_id", how="left") # third_price_df = third_df.merge(price_df, on="gujia_comm_id", how="left")
# third_price_df["ppe"] = round((third_price_df["gujia_comm_basic_prc"] - third_price_df['gujia_bds_comm_avprc']) / third_price_df['gujia_bds_comm_avprc'], 4) # third_price_df["ppe"] = round((third_price_df["gujia_comm_basic_prc"] - third_price_df['gujia_bds_comm_avprc']) / third_price_df['gujia_bds_comm_avprc'], 4)
# third_price_df = third_price_df[abs(third_price_df['ppe']) > 0.15] # third_price_df = third_price_df[abs(third_price_df['ppe']) > 0.15]
@@ -42,7 +57,7 @@ deal_price_df = deal_price_df[abs(deal_price_df['ppe']) > 0.15]
file_name = f"{month_now}测试偏差较大案例统计数据.xlsx" file_name = f"{month_now}测试偏差较大案例统计数据.xlsx"
aim_file_name = path + '/' + file_name aim_file_name = path + '/' + file_name
with pd.ExcelWriter(aim_file_name) as writer: with pd.ExcelWriter(aim_file_name) as writer:
deal_price_df.to_excel(writer, sheet_name="成交", columns=['urbn_id','urbn_nm','gujia_bds_mdl_id','gujia_admndiv_nm','gujia_comm_id','comm_nm','gujia_bds_clmtd','gujia_comm_basic_prc','gujia_bds_trdtckt_prc','ppe'], header=["城市id", "城市名称","成交id","区域","小区id","小区名称",'计算类型','计算价格','成交价格','PPE'], index=False) deal_price_df.to_excel(writer, sheet_name="成交", columns=['urbn_id','urbn_nm','gujia_bds_mdl_id','gujia_admndiv_nm','gujia_comm_id','comm_nm','gujia_bds_clmtd','gujia_comm_basic_prc','gujia_bds_trdtckt_prc','ppe','贝壳','链家','avg_price','avg_price_2b'], header=["城市id", "城市名称","成交id","区域","小区id","小区名称",'计算类型','计算价格','成交价格','PPE','贝壳均价','链家均价','房价网均价','房价网均价2b'], index=False)
# third_price_df.to_excel(writer, sheet_name="第三方", columns=['urbn_id','urbn_nm','gujia_trdpt_avprc_id','gujia_admndiv_nm','gujia_comm_id','comm_nm','gujia_bds_clmtd','gujia_comm_basic_prc','gujia_bds_comm_avprc','PPE','gujia_dtsr_dsc'], header=["城市id", "城市名称","第三方id","区域","小区id","小区名称",'计算类型','计算价格','第三方价格','PPE','来源'],index=False) # third_price_df.to_excel(writer, sheet_name="第三方", columns=['urbn_id','urbn_nm','gujia_trdpt_avprc_id','gujia_admndiv_nm','gujia_comm_id','comm_nm','gujia_bds_clmtd','gujia_comm_basic_prc','gujia_bds_comm_avprc','PPE','gujia_dtsr_dsc'], header=["城市id", "城市名称","第三方id","区域","小区id","小区名称",'计算类型','计算价格','第三方价格','PPE','来源'],index=False)
log.info("偏差较大案例统计数据结束") log.info("偏差较大案例统计数据结束")
+8 -13
View File
@@ -3,12 +3,7 @@ import logging
import pandas as pd import pandas as pd
import requests import requests
from datetime import datetime from datetime import datetime
from goojia_utils import dateutils, oracle from goojia_utils import dateutils, pgsql
from sqlalchemy import create_engine
os.environ["NLS_LANG"] = "American_America.AL32UTF8"
os.environ["ORACLE_HOME"] = "/home/airflow/instantclient_21_7"
engine = create_engine("postgresql+psycopg2://postgres:postgres@192.168.0.236:5432/warehouse", echo=False)
important_cities =["北京","上海","广州","深圳","重庆","杭州","苏州","厦门","大连","呼和浩特"] important_cities =["北京","上海","广州","深圳","重庆","杭州","苏州","厦门","大连","呼和浩特"]
city_order = pd.read_excel("/opt/airflow/dags/repo/dags/jh/para/估值模型测试城市模板.xlsx") city_order = pd.read_excel("/opt/airflow/dags/repo/dags/jh/para/估值模型测试城市模板.xlsx")
@@ -56,14 +51,14 @@ year_month = int(sys.argv[1]) if len(sys.argv) > 1 else int(dateutils.format_dat
log.info(f"测试月份:{year_month}") log.info(f"测试月份:{year_month}")
avm_evaluation_column_map = {'gujia_fsid': 'id', 'urbn_id': 'cityid', 'urbn_nm': 'cityname', 'gujia_bds_yrmo': 'year_month', 'gujia_bds_comm_num': 'community_num', 'gujiabdsvalmnycommnum': 'has_price_community_num', 'gujia_prc_cvr_rate': 'price_coverage', 'gujiabdsmnldpccommnum': 'artificial_calc_num', 'gujia_mnl_drc_prc_pct': 'artificial_calc_coverage', 'gujiabdssamp_comm_num': 'sample_calc_num', 'gujiaext_samp_clc_pct': 'sample_calc_coverage', 'gujiabdsnervetncomnum': 'ann_calc_num', 'gujianervethrunclcpct': 'ann_calc_coverage', 'gujiabds_tst_case_num': 'test_num', 'gujiabdsmdltstcommnum': 'deal_test_comm_num', 'gujiatst_comm_num_pct': 'test_comm_coverage', 'gujia_ppe15_cvr_rate': 'ppe15_coverage', 'gujia_rto_mdn': 'rate_median', 'gujia_rto_lv_san_ceft': 'coefficient', 'gujiabdsmdltstcasenum': 'deal_test_case_num', 'gujiamdltstcommnumpct': 'deal_test_comm_coverage', 'gujiamdltst_ppe15_pct': 'deal_ppe15_coverage', 'gujiamdlrtolvsan_ceft': 'deal_coefficient', 'gujia_mdl_md_bit_rto': 'deal_rate_median', 'gujiabdstrdpttcasenum': 'third_test_case_num', 'gujiabdstrdpttcommnum': 'third_test_comm_num', 'gujiatrdpttcommnumpct': 'third_test_comm_coverage', 'gujiatrdpttstppe15pct': 'third_ppe15_coverage', 'gujiatrdptrolvsanceft': 'third_coefficient', 'gujiatrdpt_md_bit_rto': 'third_rate_median', 'data_crt_tm': 'createtime', 'data_udt_tm': 'updatetime', 'gujia_src_co': 'sourcecode'} avm_evaluation_column_map = {'gujia_fsid': 'id', 'urbn_id': 'cityid', 'urbn_nm': 'cityname', 'gujia_bds_yrmo': 'year_month', 'gujia_bds_comm_num': 'community_num', 'gujiabdsvalmnycommnum': 'has_price_community_num', 'gujia_prc_cvr_rate': 'price_coverage', 'gujiabdsmnldpccommnum': 'artificial_calc_num', 'gujia_mnl_drc_prc_pct': 'artificial_calc_coverage', 'gujiabdssamp_comm_num': 'sample_calc_num', 'gujiaext_samp_clc_pct': 'sample_calc_coverage', 'gujiabdsnervetncomnum': 'ann_calc_num', 'gujianervethrunclcpct': 'ann_calc_coverage', 'gujiabds_tst_case_num': 'test_num', 'gujiabdsmdltstcommnum': 'deal_test_comm_num', 'gujiatst_comm_num_pct': 'test_comm_coverage', 'gujia_ppe15_cvr_rate': 'ppe15_coverage', 'gujia_rto_mdn': 'rate_median', 'gujia_rto_lv_san_ceft': 'coefficient', 'gujiabdsmdltstcasenum': 'deal_test_case_num', 'gujiamdltstcommnumpct': 'deal_test_comm_coverage', 'gujiamdltst_ppe15_pct': 'deal_ppe15_coverage', 'gujiamdlrtolvsan_ceft': 'deal_coefficient', 'gujia_mdl_md_bit_rto': 'deal_rate_median', 'gujiabdstrdpttcasenum': 'third_test_case_num', 'gujiabdstrdpttcommnum': 'third_test_comm_num', 'gujiatrdpttcommnumpct': 'third_test_comm_coverage', 'gujiatrdpttstppe15pct': 'third_ppe15_coverage', 'gujiatrdptrolvsanceft': 'third_coefficient', 'gujiatrdpt_md_bit_rto': 'third_rate_median', 'data_crt_tm': 'createtime', 'data_udt_tm': 'updatetime', 'gujia_src_co': 'sourcecode'}
avm_evaluation_item_column_map = {'gujia_fsid': 'id', 'urbn_id': 'cityid', 'urbn_nm': 'cityname', 'gujia_rgon_id': 'distid', 'gujia_rgon_nm': 'distname', 'gujia_comm_id': 'commid', 'gujia_comm_nm': 'commname', 'gujiabds_clc_yrmo_val': 'year_month', 'gujia_bds_clc_avprc': 'calc_avg_price', 'gujia_bds_avprc': 'avg_price', 'gujia_bds_clc_tp': 'calc_type', 'gujia_bds_rlb': 'confidence', 'gujiabds_to_mkprc_mny': 'target_avg_price', 'mdl_dt': 'dealdate', 'gujiabds_valt_unitprc': 'unit_price', 'gujia_area': 'buildarea', 'gujia_dcrt_val': 'fitment', 'gujia_twrd_val': 'houseaspect', 'gujia_bds_floor_num': 'floornum', 'gujiabdstot_floor_num': 'floorcnt', 'gujia_data_tp': 'data_type', 'gujia_rto': 'rate', 'gujia_mns_rto': 'pe', 'data_crt_tm': 'createtime'} avm_evaluation_item_column_map = {'gujia_fsid': 'id', 'urbn_id': 'cityid', 'urbn_nm': 'cityname', 'gujia_rgon_id': 'distid', 'gujia_rgon_nm': 'distname', 'gujia_comm_id': 'commid', 'gujia_comm_nm': 'commname', 'gujiabds_clc_yrmo_val': 'year_month', 'gujia_bds_clc_avprc': 'calc_avg_price', 'gujia_bds_avprc': 'avg_price', 'gujia_bds_clc_tp': 'calc_type', 'gujia_bds_rlb': 'confidence', 'gujiabds_to_mkprc_mny': 'target_avg_price', 'mdl_dt': 'dealdate', 'gujiabds_valt_unitprc': 'unit_price', 'gujia_area': 'buildarea', 'gujia_dcrt_val': 'fitment', 'gujia_twrd_val': 'houseaspect', 'gujia_bds_floor_num': 'floornum', 'gujiabdstot_floor_num': 'floorcnt', 'gujia_data_tp': 'data_type', 'gujia_rto': 'rate', 'gujia_mns_rto': 'pe', 'data_crt_tm': 'createtime'}
avm_evaluation = pd.read_sql(f"SELECT * FROM t_app_p9pha_plda.avm_evaluation_gj_h where gujia_bds_yrmo={year_month}", con=engine) avm_evaluation = pgsql.scan_by_sql_to_data_frame(f"SELECT * FROM price_stat.avm_evaluation_gj_h where gujia_bds_yrmo={year_month}")
avm_evaluation_item = pd.read_sql(f"SELECT * FROM t_app_p9pha_plda.avm_evaluation_item_gj_h where gujiabds_clc_yrmo_val={year_month}", con=engine) avm_evaluation_item = pgsql.scan_by_sql_to_data_frame(f"SELECT * FROM price_stat.avm_evaluation_item_gj_h where gujiabds_clc_yrmo_val={year_month}")
avm_evaluation.rename(columns=avm_evaluation_column_map, inplace=True) avm_evaluation.rename(columns=avm_evaluation_column_map, inplace=True)
avm_evaluation_item.rename(columns=avm_evaluation_item_column_map, inplace=True) avm_evaluation_item.rename(columns=avm_evaluation_item_column_map, inplace=True)
community_base_price = pd.read_sql(f'''SELECT gujia_bds_id as commid,urbn_nm as cityname,gujia_admndiv_nm as distname,comm_nm as commname, community_base_price = pgsql.scan_by_sql_to_data_frame(f'''SELECT gujia_bds_id as commid,urbn_nm as cityname,gujia_admndiv_nm as distname,comm_nm as commname,
gujia_comm_basic_prc as comm_avg_price,gujia_bds_cfdc_dgr as confidence,gujia_bds_clmtd as calc_type from t_app_p9pha_plda.avm_community_base_price_gj_h where gujiabds_clc_yrmo_val={year_month}''', gujia_comm_basic_prc as comm_avg_price,gujia_bds_cfdc_dgr as confidence,gujia_bds_clmtd as calc_type from price_stat.avm_community_base_price where gujiabds_clc_yrmo_val={year_month}'''
con=engine) )
communities = pd.read_sql(f"select gujia_comm_id as COMMID from base.t0831_bigdata_community_gj_h", con=engine) communities = pgsql.scan_by_sql_to_data_frame(f"select gujia_comm_id as COMMID from base.bigdata_community")
community_base_price = community_base_price[community_base_price.commid.isin(communities["commid"])] community_base_price = community_base_price[community_base_price.commid.isin(communities["commid"])]
avm_evaluation_item["fitment"] = avm_evaluation_item["fitment"].map(convert_fitment_code) avm_evaluation_item["fitment"] = avm_evaluation_item["fitment"].map(convert_fitment_code)
@@ -96,7 +91,7 @@ import_avm_evaluation_item_third.sort_values(by="cityrank", inplace=True)
file_name = f"{year_month}估值模型测试.xlsx" file_name = f"{year_month}估值模型测试.xlsx"
aim_file_name = path + '/' + file_name aim_file_name = path + '/' + file_name
with pd.ExcelWriter(aim_file_name) as writer: with pd.ExcelWriter(aim_file_name) as writer:
avm_evaluation.to_excel(writer, sheet_name="92个城市估值概况", columns=["cityname","community_num","price_coverage","sample_calc_coverage","ann_calc_coverage","artificial_calc_coverage","deal_test_case_num","deal_ppe15_coverage","deal_test_comm_num","deal_test_comm_coverage","deal_rate_median","deal_coefficient","third_test_comm_num","third_test_comm_coverage","third_ppe15_coverage","third_rate_median","third_coefficient"], header=["城市名称","小区量","可输出均价小区占比","样本计算占比","神经网络计算占比","人工指导价占比","成交测试案例总量","成交测试PPE15占比","成交测试小区量","成交测试小区量占比","成交比率中位数","成交比率离散系数","第三方均价测试小区量","第三方均价测试小区量占比","第三方均价测试PPE15占比","第三方比率中位数","第三方比率离散系数"], index=False) avm_evaluation.to_excel(writer, encoding="UTF-8", sheet_name="92个城市估值概况", columns=["cityname","community_num","price_coverage","sample_calc_coverage","ann_calc_coverage","artificial_calc_coverage","deal_test_case_num","deal_ppe15_coverage","deal_test_comm_num","deal_test_comm_coverage","deal_rate_median","deal_coefficient","third_test_comm_num","third_test_comm_coverage","third_ppe15_coverage","third_rate_median","third_coefficient"], header=["城市名称","小区量","可输出均价小区占比","样本计算占比","神经网络计算占比","人工指导价占比","成交测试案例总量","成交测试PPE15占比","成交测试小区量","成交测试小区量占比","成交比率中位数","成交比率离散系数","第三方均价测试小区量","第三方均价测试小区量占比","第三方均价测试PPE15占比","第三方比率中位数","第三方比率离散系数"], index=False)
import_community_base_price.to_excel(writer, sheet_name="测试10个城市估值数据", columns=["cityname","distname","commname","comm_avg_price","confidence","calc_type"], header=["城市","区域","小区名","估值小区均价","置信度","估值方式"],index=False) import_community_base_price.to_excel(writer, sheet_name="测试10个城市估值数据", columns=["cityname","distname","commname","comm_avg_price","confidence","calc_type"], header=["城市","区域","小区名","估值小区均价","置信度","估值方式"],index=False)
import_avm_evaluation_item_deal.to_excel(writer, sheet_name="10个城市成交测试案例", columns=['cityname','distname', 'commname', 'unit_price', 'confidence', 'calc_type','total_price', 'deal_total_price', 'target_avg_price','pe',"dealdate","fitment","houseaspect","floornum","floorcnt","buildarea"], header=["城市","区域","小区名","修正均价","置信度","估值方式","小区估价","对标成交价","对标均价","PPE","dealdate","fitment","houseaspect","floornum","floorcnt","buildarea"], index=False) import_avm_evaluation_item_deal.to_excel(writer, sheet_name="10个城市成交测试案例", columns=['cityname','distname', 'commname', 'unit_price', 'confidence', 'calc_type','total_price', 'deal_total_price', 'target_avg_price','pe',"dealdate","fitment","houseaspect","floornum","floorcnt","buildarea"], header=["城市","区域","小区名","修正均价","置信度","估值方式","小区估价","对标成交价","对标均价","PPE","dealdate","fitment","houseaspect","floornum","floorcnt","buildarea"], index=False)
import_avm_evaluation_item_third.to_excel(writer, sheet_name="10个城市第三方均价测试", columns=["cityname","distname","commname","avg_price","confidence","calc_type","target_avg_price","pe"], header=["城市","区域","小区名","小区均价","置信度","估值方式","对标第三方均价","PPE"],index=False) import_avm_evaluation_item_third.to_excel(writer, sheet_name="10个城市第三方均价测试", columns=["cityname","distname","commname","avg_price","confidence","calc_type","target_avg_price","pe"], header=["城市","区域","小区名","小区均价","置信度","估值方式","对标第三方均价","PPE"],index=False)
+17 -13
View File
@@ -1,16 +1,19 @@
""" """
价格异常检测脚本 价格异常检测脚本
""" """
from goojia_utils import dateutils, oracle from goojia_utils import dateutils, pgsql, MAIN_MONGO_CLIENT, data_utils
import pandas as pd import pandas as pd
import logging import logging
import os import os
import sys import sys
import requests import requests
from sqlalchemy import create_engine
os.environ["ORACLE_HOME"] = "/home/airflow/instantclient_21_7" seaweed_conn = MAIN_MONGO_CLIENT.get_collection('fangjia', 'seaweed')
engine = create_engine("postgresql+psycopg2://postgres:postgres@192.168.0.236:5432/warehouse", echo=False) # 获取独家小区
exclusive_seaweed_df = pd.DataFrame(list(seaweed_conn.find({'cat': 'district', 'status': 0, 'mark_map.exclusive': '是'}, {'city': 1, 'region': 1, 'name': 1, 'mark_map.exclusive': 1}).batch_size(100000)))
exclusive_seaweed_df = data_utils.flatten_json(exclusive_seaweed_df, 'mark_map')
exclusive_seaweed_df = exclusive_seaweed_df.rename(columns={'name': 'commname', 'exclusive': '独家小区'})
def simple_calc(result_df): def simple_calc(result_df):
""" """
新增是否有样本列 新增是否有样本列
@@ -26,9 +29,9 @@ def simple_calc(result_df):
if len(actions) == 800: if len(actions) == 800:
ins = "','".join(actions) ins = "','".join(actions)
ins = "'" + ins + "'" ins = "'" + ins + "'"
simple_temp_df = pd.read_sql( simple_temp_df = pgsql.scan_by_sql_to_data_frame(
"select distinct gujia_comm_id as commid from base.t0831_bigdata_forsale_gj_h where " + "select distinct gujia_comm_id as commid from base.bigdata_forsale_gj_h where " +
" anc_tm> TO_DATE('" + month_before + "','YYYY-MM-DD') and gujia_comm_id in (" + ins + ")", con=engine) " anc_tm> TO_DATE('" + month_before + "','YYYY-MM-DD') and gujia_comm_id in (" + ins + ")")
simple_df = pd.concat([simple_df, simple_temp_df]) simple_df = pd.concat([simple_df, simple_temp_df])
actions.clear() actions.clear()
@@ -36,9 +39,9 @@ def simple_calc(result_df):
if len(actions) > 0: if len(actions) > 0:
ins = "','".join(actions) ins = "','".join(actions)
ins = "'" + ins + "'" ins = "'" + ins + "'"
simple_temp_df = pd.read_sql( simple_temp_df = pgsql.scan_by_sql_to_data_frame(
"select distinct gujia_comm_id as commid from base.t0831_bigdata_forsale_gj_h where " + "select distinct gujia_comm_id as commid from base.bigdata_forsale_gj_h where " +
" anc_tm> TO_DATE('" + month_before + "','YYYY-MM-DD') and gujia_comm_id in (" + ins + ")", con=engine) " anc_tm> TO_DATE('" + month_before + "','YYYY-MM-DD') and gujia_comm_id in (" + ins + ")")
simple_df = pd.concat([simple_df, simple_temp_df]) simple_df = pd.concat([simple_df, simple_temp_df])
actions.clear() actions.clear()
simple_df['has_sample'] = 1 simple_df['has_sample'] = 1
@@ -88,8 +91,8 @@ if type == month_last:
month_now = month_last month_now = month_last
month_last = month_before month_last = month_before
df = pd.read_sql("SELECT gujia_bss_prc_id as id, urbn_nm as cityname, gujia_admndiv_nm as distname, gujia_bds_id as commid, comm_nm as commname, gujia_bds_clmtd as calc_type, gujia_bds_cfdc_dgr as confidence, gujia_comm_basic_prc as avg_price, gujiabds_clc_yrmo_val as year_month FROM t_app_p9pha_plda.avm_community_base_price_gj_h WHERE gujiabds_clc_yrmo_val IN('"+month_now+"', '"+month_last+"') " + df = pgsql.scan_by_sql_to_data_frame("SELECT gujia_bss_prc_id as id, urbn_nm as cityname, gujia_admndiv_nm as distname, gujia_bds_id as commid, comm_nm as commname, gujia_bds_clmtd as calc_type, gujia_bds_cfdc_dgr as confidence, gujia_comm_basic_prc as avg_price, gujiabds_clc_yrmo_val as year_month FROM price_stat.avm_community_base_price WHERE gujiabds_clc_yrmo_val IN('"+month_now+"', '"+month_last+"') " +
" ORDER BY urbn_nm, gujia_admndiv_nm, gujia_bds_id, comm_nm, gujia_bds_clmtd", con=engine) " ORDER BY urbn_nm, gujia_admndiv_nm, gujia_bds_id, comm_nm, gujia_bds_clmtd")
six_map = dict() six_map = dict()
five_map = dict() five_map = dict()
@@ -113,7 +116,7 @@ for key in six_map:
if abs(six_price - five_price) * 100 / five_price > 10: if abs(six_price - five_price) * 100 / five_price > 10:
actions.append({'city': info[0], 'region': info[1], 'commid': info[2], 'commname': info[3], actions.append({'city': info[0], 'region': info[1], 'commid': info[2], 'commname': info[3],
'five_price': five_price,'five_calc_type': five_map[key].split("|")[1], 'five_price': five_price,'five_calc_type': five_map[key].split("|")[1],
'six_price': six_price,'six_calc_type': six_map[key].split("|")[1], 'six_price': six_price,'six_calc_type': six_map[key].split("|")[1], '差异值': abs(six_price - five_price) * 100 / five_price,
'涨跌幅': (six_price - five_price) * 100 / five_price, 'confidence': six_map[key].split("|")[2]}) '涨跌幅': (six_price - five_price) * 100 / five_price, 'confidence': six_map[key].split("|")[2]})
if info[0] not in city_seaweed_dict: if info[0] not in city_seaweed_dict:
city_seaweed_dict[info[0]] = 1 city_seaweed_dict[info[0]] = 1
@@ -134,6 +137,7 @@ excel_writer = pd.ExcelWriter(aim_file_name)
if type == 'sample': if type == 'sample':
result_df = simple_calc(result_df) result_df = simple_calc(result_df)
result_df = pd.merge(result_df, exclusive_seaweed_df, on=['city', 'region', 'commname'], how='left')
result_df = result_df.drop_duplicates(subset=['commid'], keep='first') result_df = result_df.drop_duplicates(subset=['commid'], keep='first')
result_df = result_df.rename(columns={'city': '城市', 'region': '区域', 'commid': '小区ID', 'commname': '小区名', 'five_price': '上一月价格', result_df = result_df.rename(columns={'city': '城市', 'region': '区域', 'commid': '小区ID', 'commname': '小区名', 'five_price': '上一月价格',
'five_calc_type': '上一月计算方式', 'six_price': '当月价格', 'six_calc_type': '当月计算方式', 'confidence': '置信度'}) 'five_calc_type': '上一月计算方式', 'six_price': '当月价格', 'six_calc_type': '当月计算方式', 'confidence': '置信度'})

Some files were not shown because too many files have changed in this diff Show More